[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:07 dist-test-slave-9tkz krb5kdc[8160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:07 dist-test-slave-9tkz krb5kdc[8160](info): set up 2 sockets
Jun 19 02:44:07 dist-test-slave-9tkz krb5kdc[8160](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:44:08.313774 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:08 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:08.487582 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41027
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:08.820170 8173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:08.820552 8173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:08.820725 8173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:08.828505 8173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:08.828694 8173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:08.828806 8173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:08.828907 8173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:08.828989 8173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:08.845254 8173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:08.846848 8173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:08.848873 8173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:08.863502 8181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:08.863559 8179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:08.863538 8178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:08.864971 8173 server_base.cc:1034] running on GCE node
I20260619 02:44:08.866123 8173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:08.868191 8173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:08.869459 8173 hybrid_clock.cc:648] HybridClock initialized: now 1781837048869392 us; error 51 us; skew 500 ppm
I20260619 02:44:08.869963 8173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:08 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837048, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:08.874307 8173 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:08.876075 8173 webserver.cc:460] Webserver started at http://127.7.246.126:40649/ using document root <none> and password file <none>
I20260619 02:44:08.877010 8173 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:08.877149 8173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:08.877485 8173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:08.881949 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "90b094aecd6347d8ace141e62ba72942"
format_stamp: "Formatted at 2026-06-19 02:44:08 on dist-test-slave-9tkz"
server_key: "723d931c7ec67887fca8a6224f1010bc"
server_key_iv: "11e29bacf359b6efbe141974b26d01bd"
server_key_version: "encryptionkey@0"
I20260619 02:44:08.882704 8173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "90b094aecd6347d8ace141e62ba72942"
format_stamp: "Formatted at 2026-06-19 02:44:08 on dist-test-slave-9tkz"
server_key: "723d931c7ec67887fca8a6224f1010bc"
server_key_iv: "11e29bacf359b6efbe141974b26d01bd"
server_key_version: "encryptionkey@0"
I20260619 02:44:08.888059 8173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 02:44:08.891541 8188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:08.892974 8173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:08.893121 8173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "90b094aecd6347d8ace141e62ba72942"
format_stamp: "Formatted at 2026-06-19 02:44:08 on dist-test-slave-9tkz"
server_key: "723d931c7ec67887fca8a6224f1010bc"
server_key_iv: "11e29bacf359b6efbe141974b26d01bd"
server_key_version: "encryptionkey@0"
I20260619 02:44:08.893280 8173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:08.913951 8173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:08.917949 8173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:08.918174 8173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:08.937281 8173 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41027
I20260619 02:44:08.937316 8239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41027 every 8 connection(s)
I20260619 02:44:08.938681 8173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:08.942898 8240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:08.944664 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8173
I20260619 02:44:08.944993 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:08.947443 8153 external_mini_cluster.cc:1428] Setting key 5817b93654ec52add6828c08653a3a96
I20260619 02:44:08.955858 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942: Bootstrap starting.
I20260619 02:44:08.958842 8240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:08.959813 8240 log.cc:824] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:08.963001 8240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942: No bootstrap required, opened a new log
I20260619 02:44:08.972386 8240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } }
I20260619 02:44:08.972852 8240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:08.972980 8240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b094aecd6347d8ace141e62ba72942, State: Initialized, Role: FOLLOWER
I20260619 02:44:08.973698 8240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } }
I20260619 02:44:08.973959 8240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:08.974112 8240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:08.974287 8240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:08.976617 8240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } }
I20260619 02:44:08.977142 8240 leader_election.cc:302] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b094aecd6347d8ace141e62ba72942; no voters:
I20260619 02:44:08.977802 8240 leader_election.cc:288] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 19 02:44:08 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:08.977900 8245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:08.979460 8245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [term 1 LEADER]: Becoming Leader. State: Replica: 90b094aecd6347d8ace141e62ba72942, State: Running, Role: LEADER
I20260619 02:44:08.980108 8245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } }
I20260619 02:44:08.980765 8240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:08.983765 8246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b094aecd6347d8ace141e62ba72942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } } }
I20260619 02:44:08.984184 8246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:08.985996 8247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b094aecd6347d8ace141e62ba72942. Latest consensus state: current_term: 1 leader_uuid: "90b094aecd6347d8ace141e62ba72942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b094aecd6347d8ace141e62ba72942" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41027 } } }
I20260619 02:44:08.986346 8247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:08.987706 8251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:08.991971 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:08.952078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.7.246.126:41027)
0619 02:44:08.953008 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:08.953028 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:08.954336 (+ 1308us) server_negotiation.cc:407] Connection header received
0619 02:44:08.964470 (+ 10134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:08.964498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:08.964963 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:08.965425 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:08.968525 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:08.970617 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:08.972255 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:08.973299 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:08.973484 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:08.980019 (+ 6535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:08.980027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:08.980042 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:08.980142 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:08.985099 (+ 4957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:08.986804 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:08.986816 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:08.986831 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:08.987456 (+ 625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:08.988056 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:08.988062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:08.988065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:08.988803 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:08.989132 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:08.990254 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0619 02:44:08.990649 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":412,"threads_started":1}
I20260619 02:44:08.997614 8251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:09.006559 8251 catalog_manager.cc:1347] Generated new cluster ID: 76ca2d5a0c07420daca017659a2c344c
I20260619 02:44:09.006690 8251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:09.027894 8251 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:09.030272 8251 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:09.046061 8251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90b094aecd6347d8ace141e62ba72942: Generated new TSK 0
I20260619 02:44:09.047202 8251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:09.096499 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:09.436610 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:09.436904 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:09.437000 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:09.444363 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:09.444484 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:09.444602 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:09.454859 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:09.456198 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:09.457736 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:09.471652 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:09.471777 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:09.471777 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:09.472685 8266 server_base.cc:1034] running on GCE node
I20260619 02:44:09.473320 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:09.474442 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:09.475668 8266 hybrid_clock.cc:648] HybridClock initialized: now 1781837049475613 us; error 54 us; skew 500 ppm
I20260619 02:44:09.476028 8266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:09 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837049, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:09.479341 8266 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:09.480959 8266 webserver.cc:460] Webserver started at http://127.7.246.65:40857/ using document root <none> and password file <none>
I20260619 02:44:09.481834 8266 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:09.481961 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:09.482301 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:09.485222 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "f345078124b94e90a574030610364439"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "304fd19fe0b12ed5d9049ca91342353d"
server_key_iv: "5920a0827cce77e3b1007b94d2f405b6"
server_key_version: "encryptionkey@0"
I20260619 02:44:09.486052 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "f345078124b94e90a574030610364439"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "304fd19fe0b12ed5d9049ca91342353d"
server_key_iv: "5920a0827cce77e3b1007b94d2f405b6"
server_key_version: "encryptionkey@0"
I20260619 02:44:09.491575 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:09.495457 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:09.496961 8266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:44:09.497670 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "f345078124b94e90a574030610364439"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "304fd19fe0b12ed5d9049ca91342353d"
server_key_iv: "5920a0827cce77e3b1007b94d2f405b6"
server_key_version: "encryptionkey@0"
I20260619 02:44:09.498116 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:09.517105 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:09.521169 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:09.521487 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:09.522481 8266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:09.523921 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:09.524026 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:09.524175 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:09.524245 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:09.558801 8266 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36969
I20260619 02:44:09.558837 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36969 every 8 connection(s)
I20260619 02:44:09.560093 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:09.564280 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8266
I20260619 02:44:09.564488 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:09.565076 8153 external_mini_cluster.cc:1428] Setting key 1a65fbb5ca9b04fff32eb68339681f17
Jun 19 02:44:09 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837049, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:09.583984 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:09.562638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:41753 (local address 127.7.246.126:41027)
0619 02:44:09.563062 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:09.563066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:09.565154 (+ 2088us) server_negotiation.cc:407] Connection header received
0619 02:44:09.567294 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:09.567300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:09.567388 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:09.567508 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:09.569126 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:09.570345 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:09.572127 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:09.572874 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:09.573017 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:09.576689 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:09.576698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:09.576706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:09.576753 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:09.580456 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:09.581405 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:09.581413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:09.581419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:09.581589 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:09.582047 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:09.582052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:09.582056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:09.582471 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:09.582591 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:09.583372 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 02:44:09.583598 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:44:09.584992 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:09.563273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41027 (local address 127.7.246.65:41753)
0619 02:44:09.564973 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:09.565020 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:09.567032 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:09.567762 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:09.567770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:09.568332 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:09.568928 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:09.568945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:09.570489 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:09.570497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:09.571999 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:09.572011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:09.573104 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:09.573109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:09.573338 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:09.574078 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:09.574105 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:09.576437 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:09.580640 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:09.580648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:09.580672 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:09.581214 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:09.581713 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:09.581717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:09.581720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:09.581930 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:09.582604 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:09.582611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:09.582972 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 02:44:09.583269 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 02:44:09.583841 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":1230,"threads_started":1}
I20260619 02:44:09.586651 8394 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41027
I20260619 02:44:09.587034 8394 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:09.587941 8394 heartbeater.cc:511] Master 127.7.246.126:41027 requested a full tablet report, sending...
I20260619 02:44:09.590276 8205 ts_manager.cc:194] Registered new tserver with Master: f345078124b94e90a574030610364439 (127.7.246.65:36969)
I20260619 02:44:09.591727 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:41753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:09.604884 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:09.942795 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:09.943110 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:09.943285 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:09.950708 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:09.950866 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:09.951049 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:09.964495 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:09.966058 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:09.967424 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:09.980844 8405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:09.980777 8406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:09.980821 8408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:09.981751 8400 server_base.cc:1034] running on GCE node
I20260619 02:44:09.982460 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:09.983392 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:09.984594 8400 hybrid_clock.cc:648] HybridClock initialized: now 1781837049984533 us; error 55 us; skew 500 ppm
I20260619 02:44:09.984951 8400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:09 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837049, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:09.988399 8400 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:09.990444 8400 webserver.cc:460] Webserver started at http://127.7.246.66:39141/ using document root <none> and password file <none>
I20260619 02:44:09.991117 8400 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:09.991214 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:09.991468 8400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:09.994498 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "88e49f8b60f14bc394fb36daafe73a6d"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "cdc9c0343fa2e771916ef7bf4681ee6f"
server_key_iv: "cd23997df9f71e98f1e5bdfe667d2b0f"
server_key_version: "encryptionkey@0"
I20260619 02:44:09.995338 8400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "88e49f8b60f14bc394fb36daafe73a6d"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "cdc9c0343fa2e771916ef7bf4681ee6f"
server_key_iv: "cd23997df9f71e98f1e5bdfe667d2b0f"
server_key_version: "encryptionkey@0"
I20260619 02:44:10.000672 8400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 02:44:10.004309 8415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.005715 8400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:10.005861 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "88e49f8b60f14bc394fb36daafe73a6d"
format_stamp: "Formatted at 2026-06-19 02:44:09 on dist-test-slave-9tkz"
server_key: "cdc9c0343fa2e771916ef7bf4681ee6f"
server_key_iv: "cd23997df9f71e98f1e5bdfe667d2b0f"
server_key_version: "encryptionkey@0"
I20260619 02:44:10.006019 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:10.021018 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:10.025120 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:10.025360 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:10.026755 8400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:10.028007 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:10.028075 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.028160 8400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:10.028204 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.063668 8400 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:46267
I20260619 02:44:10.063782 8527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:46267 every 8 connection(s)
I20260619 02:44:10.064934 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:10.072259 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8400
I20260619 02:44:10.072463 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:10.072911 8153 external_mini_cluster.cc:1428] Setting key e7e3ea1e1588cd5bbb44dd956cabc445
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837049, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:10.086301 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.067462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:45025 (local address 127.7.246.126:41027)
0619 02:44:10.067629 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.067633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.068747 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 02:44:10.070626 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.070631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.070722 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.070843 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.072501 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.073298 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.075302 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.075722 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.075827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.080764 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.080770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.080773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.080806 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.083442 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.084144 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.084149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.084151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.084274 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.084679 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.084684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.084687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.084950 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.085100 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.085887 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.086066 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:44:10.087909 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.067931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41027 (local address 127.7.246.66:45025)
0619 02:44:10.068591 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:10.068632 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:10.070369 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:10.071077 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:10.071084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:10.071592 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:10.072325 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:10.072353 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.073456 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.073460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:10.075146 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:10.075161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.075949 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.075958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:10.076316 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.077068 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:10.077105 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:10.080509 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:10.083577 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:10.083586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:10.083603 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:10.084019 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:10.084389 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:10.084394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:10.084397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:10.084570 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:10.085063 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:10.085071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:10.085416 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 02:44:10.086524 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0619 02:44:10.086903 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":251,"threads_started":1}
I20260619 02:44:10.089380 8528 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41027
I20260619 02:44:10.089785 8528 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:44:10.090776 8528 heartbeater.cc:511] Master 127.7.246.126:41027 requested a full tablet report, sending...
I20260619 02:44:10.092212 8205 ts_manager.cc:194] Registered new tserver with Master: 88e49f8b60f14bc394fb36daafe73a6d (127.7.246.66:46267)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:44:10.093118 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:45025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:10.113597 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:10.448690 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:10.448998 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:10.449162 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:10.456491 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:10.456611 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:10.456818 8533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:10.467177 8533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41027
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:10.468473 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:10.469935 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:10.483808 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:10.483866 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:10.483951 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:10.485132 8533 server_base.cc:1034] running on GCE node
I20260619 02:44:10.485832 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:10.486847 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:10.488045 8533 hybrid_clock.cc:648] HybridClock initialized: now 1781837050487983 us; error 59 us; skew 500 ppm
I20260619 02:44:10.488380 8533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:10.491575 8533 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:10.493083 8533 webserver.cc:460] Webserver started at http://127.7.246.67:45179/ using document root <none> and password file <none>
I20260619 02:44:10.493870 8533 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:10.494001 8533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:10.494321 8533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:10.497340 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "8d7e3d9a73fb4497974a8671fe17e44d"
format_stamp: "Formatted at 2026-06-19 02:44:10 on dist-test-slave-9tkz"
server_key: "1caee709593c831fd9180109df1d178f"
server_key_iv: "09219385d32df9ece9998633a6aba5c2"
server_key_version: "encryptionkey@0"
I20260619 02:44:10.498736 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "8d7e3d9a73fb4497974a8671fe17e44d"
format_stamp: "Formatted at 2026-06-19 02:44:10 on dist-test-slave-9tkz"
server_key: "1caee709593c831fd9180109df1d178f"
server_key_iv: "09219385d32df9ece9998633a6aba5c2"
server_key_version: "encryptionkey@0"
I20260619 02:44:10.504168 8533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:10.507834 8548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.509250 8533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:10.509421 8533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "8d7e3d9a73fb4497974a8671fe17e44d"
format_stamp: "Formatted at 2026-06-19 02:44:10 on dist-test-slave-9tkz"
server_key: "1caee709593c831fd9180109df1d178f"
server_key_iv: "09219385d32df9ece9998633a6aba5c2"
server_key_version: "encryptionkey@0"
I20260619 02:44:10.509666 8533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:10.526922 8533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:10.531190 8533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:10.531438 8533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:10.532790 8533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:10.534154 8533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:10.534230 8533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.534325 8533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:10.534368 8533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:10.568588 8533 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:38439
I20260619 02:44:10.568632 8660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:38439 every 8 connection(s)
I20260619 02:44:10.569940 8533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:10.580243 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8533
I20260619 02:44:10.580440 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:10.581032 8153 external_mini_cluster.cc:1428] Setting key 3684cd237316a935f3322b23f5373da5
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:10.592245 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.572423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:53095 (local address 127.7.246.126:41027)
0619 02:44:10.572593 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.572598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.574314 (+ 1716us) server_negotiation.cc:407] Connection header received
0619 02:44:10.576075 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.576080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.576159 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.576258 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.577748 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.578519 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.580046 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.580509 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.580613 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.585488 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.585497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.585527 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.585608 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.589060 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.589915 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.589920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.589924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.590060 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.590486 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.590491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.590494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.590762 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.590939 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.591652 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.591862 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:44:10.593214 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.572897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41027 (local address 127.7.246.67:53095)
0619 02:44:10.574137 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:10.574189 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:10.575818 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:10.576494 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:10.576503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:10.577009 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:10.577586 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:10.577604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.578649 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.578653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:10.579913 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:10.579927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.580736 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.580743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:10.581042 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.582019 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:10.582053 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:10.585237 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:10.589222 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:10.589231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:10.589251 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:10.589787 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:10.590171 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:10.590176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:10.590180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:10.590368 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:10.590919 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:10.590927 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:10.591301 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 02:44:10.591566 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 02:44:10.592107 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":247,"threads_started":1}
I20260619 02:44:10.594697 8661 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41027
I20260619 02:44:10.594753 8394 heartbeater.cc:503] Master 127.7.246.126:41027 was elected leader, sending a full tablet report...
I20260619 02:44:10.594997 8661 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:10.595824 8661 heartbeater.cc:511] Master 127.7.246.126:41027 requested a full tablet report, sending...
I20260619 02:44:10.597473 8205 ts_manager.cc:194] Registered new tserver with Master: 8d7e3d9a73fb4497974a8671fe17e44d (127.7.246.67:38439)
I20260619 02:44:10.598255 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:10.598519 8205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:53095
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:10.643250 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.628127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.7.246.65:36969)
0619 02:44:10.628737 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.628749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.629174 (+ 425us) server_negotiation.cc:407] Connection header received
0619 02:44:10.629377 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.629387 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.629673 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.629837 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.631248 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.632245 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.633501 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.634109 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.634238 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.638073 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.638085 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.638101 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.638139 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.640931 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.641504 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.641512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.641514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.641682 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.642016 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.642019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.642021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.642340 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.642562 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.642778 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.642972 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
W20260619 02:44:10.644474 8309 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42656
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:10.687505 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.673795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.7.246.65:36969)
0619 02:44:10.674012 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.674018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.674204 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:44:10.674359 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.674363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.674424 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.674506 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.675996 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.676666 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.678001 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.678416 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.678540 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.682455 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.682475 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.682478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.682510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.685352 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.685989 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.685993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.685995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.686100 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.686375 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.686377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.686379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.686612 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.686777 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.687036 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.687184 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:44:10.689031 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8266
I20260619 02:44:10.701400 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.689974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.7.246.65:36969)
0619 02:44:10.690115 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.690119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.690444 (+ 325us) server_negotiation.cc:407] Connection header received
0619 02:44:10.690544 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.690547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.690604 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.690709 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.691787 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.692453 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.694112 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.694533 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.694624 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.696646 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.696651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.696655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.696691 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.699451 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.699993 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.699996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.699998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.700116 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.700381 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.700384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.700386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.700607 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.700774 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.700997 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.701133 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:44:10.702139 8389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:10.784251 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8266
I20260619 02:44:10.800689 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8400
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:10.816025 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.801595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.7.246.66:46267)
0619 02:44:10.802085 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.802093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.802117 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:44:10.802176 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.802184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.802410 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.802541 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.803583 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.804640 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.805899 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.806458 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.806563 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.810446 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.810458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.810473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.810509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.813472 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.814161 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.814169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.814171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.814311 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.814647 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.814651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.814652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.815011 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.815230 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.815451 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.815646 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260619 02:44:10.817225 8523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:10.898895 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8400
I20260619 02:44:10.920610 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8533
Jun 19 02:44:10 dist-test-slave-9tkz krb5kdc[8160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:10.935809 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:10.921642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.7.246.67:38439)
0619 02:44:10.922284 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:10.922291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:10.922320 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:44:10.922380 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:10.922386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:10.922589 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:10.922728 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:10.923845 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.924836 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.926079 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:10.926610 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:10.926699 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:10.930541 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:10.930552 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:10.930565 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:10.930597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:10.933052 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.933810 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.933817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.933819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.933931 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:10.934288 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:10.934297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:10.934299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:10.934610 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:10.934907 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:10.935210 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:44:10.935481 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":335,"threads_started":1}
I20260619 02:44:10.937074 8656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:11.017894 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8533
I20260619 02:44:11.033879 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8173
I20260619 02:44:11.035039 8235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:11.135264 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8173
2026-06-19T02:44:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3914 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:11 dist-test-slave-9tkz krb5kdc[8682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:11 dist-test-slave-9tkz krb5kdc[8682](info): set up 2 sockets
Jun 19 02:44:11 dist-test-slave-9tkz krb5kdc[8682](info): commencing operation
krb5kdc: starting...
W20260619 02:44:12.209429 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:12 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:12.382576 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:46773
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:46773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:12.707266 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:12.707577 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:12.707722 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:12.714919 8695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:12.715039 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:12.715153 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:12.715248 8695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:12.715327 8695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:12.725298 8695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:46773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:12.726724 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:12.728050 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:12.739916 8701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:12.740041 8703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:12.739986 8700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:12.741017 8695 server_base.cc:1034] running on GCE node
I20260619 02:44:12.741905 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:12.743341 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:12.744572 8695 hybrid_clock.cc:648] HybridClock initialized: now 1781837052744528 us; error 37 us; skew 500 ppm
I20260619 02:44:12.744915 8695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:12 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837052, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:12.752766 8695 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:12.754441 8695 webserver.cc:460] Webserver started at http://127.7.246.126:37277/ using document root <none> and password file <none>
I20260619 02:44:12.755174 8695 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:12.755318 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:12.755630 8695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:12.758754 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "85774a645caa44358126eee7b34f2545"
format_stamp: "Formatted at 2026-06-19 02:44:12 on dist-test-slave-9tkz"
server_key: "69d3486c5e1277d5116f6061af9ec7a8"
server_key_iv: "b319c938f10e4508dda5ee84f99e1663"
server_key_version: "encryptionkey@0"
I20260619 02:44:12.759523 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "85774a645caa44358126eee7b34f2545"
format_stamp: "Formatted at 2026-06-19 02:44:12 on dist-test-slave-9tkz"
server_key: "69d3486c5e1277d5116f6061af9ec7a8"
server_key_iv: "b319c938f10e4508dda5ee84f99e1663"
server_key_version: "encryptionkey@0"
I20260619 02:44:12.764714 8695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:12.768193 8710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:12.769517 8695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:12.769747 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "85774a645caa44358126eee7b34f2545"
format_stamp: "Formatted at 2026-06-19 02:44:12 on dist-test-slave-9tkz"
server_key: "69d3486c5e1277d5116f6061af9ec7a8"
server_key_iv: "b319c938f10e4508dda5ee84f99e1663"
server_key_version: "encryptionkey@0"
I20260619 02:44:12.769970 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:12.790580 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:12.795610 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:12.795909 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:12.814293 8695 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:46773
I20260619 02:44:12.814301 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:46773 every 8 connection(s)
I20260619 02:44:12.815624 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:12.819308 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8695
I20260619 02:44:12.819552 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:12.819555 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:12.820374 8153 external_mini_cluster.cc:1428] Setting key 43f9624674385dff3b454a4b85b4ed82
I20260619 02:44:12.833385 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: Bootstrap starting.
Jun 19 02:44:12 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:12.836723 8762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:12.838104 8762 log.cc:824] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:12.841511 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: No bootstrap required, opened a new log
I20260619 02:44:12.844683 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:12.822500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.7.246.126:46773)
0619 02:44:12.823329 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:12.823340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:12.823380 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:44:12.825681 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:12.825746 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:12.826283 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:12.826769 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:12.827969 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:12.830009 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:12.831220 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:12.832310 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:12.832472 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:12.836252 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:12.836266 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:12.836288 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:12.836331 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:12.840027 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:12.840693 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:12.840711 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:12.840719 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:12.840919 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:12.841234 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:12.841238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:12.841241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:12.842051 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:12.842349 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:12.842841 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 02:44:12.843273 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":419,"threads_started":1}
I20260619 02:44:12.850764 8762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } }
I20260619 02:44:12.851127 8762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:12.851191 8762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85774a645caa44358126eee7b34f2545, State: Initialized, Role: FOLLOWER
I20260619 02:44:12.851768 8762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } }
I20260619 02:44:12.851969 8762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:12.852113 8762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:12.852249 8762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:12.855111 8762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } }
I20260619 02:44:12.855590 8762 leader_election.cc:302] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85774a645caa44358126eee7b34f2545; no voters:
I20260619 02:44:12.856104 8762 leader_election.cc:288] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:12.856251 8767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:12.856947 8767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [term 1 LEADER]: Becoming Leader. State: Replica: 85774a645caa44358126eee7b34f2545, State: Running, Role: LEADER
I20260619 02:44:12.857475 8767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } }
I20260619 02:44:12.858367 8762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:12.860737 8768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85774a645caa44358126eee7b34f2545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } } }
I20260619 02:44:12.861128 8768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:12.861527 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85774a645caa44358126eee7b34f2545. Latest consensus state: current_term: 1 leader_uuid: "85774a645caa44358126eee7b34f2545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85774a645caa44358126eee7b34f2545" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46773 } } }
I20260619 02:44:12.861982 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545 [sys.catalog]: This master's current role is: LEADER
W20260619 02:44:12.872882 8783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:44:12.873075 8783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:44:12.873222 8774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:12.875419 8774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:12.883975 8774 catalog_manager.cc:1347] Generated new cluster ID: cace13232b0740ff881932eb215fe6a1
I20260619 02:44:12.884092 8774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:12.922294 8774 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:12.923736 8774 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:12.931948 8774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85774a645caa44358126eee7b34f2545: Generated new TSK 0
I20260619 02:44:12.932735 8774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:12.972687 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:13.292744 8788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:13.293030 8788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:13.293186 8788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:13.300674 8788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:13.300791 8788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:13.300954 8788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:13.311066 8788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:13.312377 8788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:13.313701 8788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:13.327315 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:13.327298 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:13.327833 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:13.328114 8788 server_base.cc:1034] running on GCE node
I20260619 02:44:13.328825 8788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:13.329917 8788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:13.331182 8788 hybrid_clock.cc:648] HybridClock initialized: now 1781837053331108 us; error 47 us; skew 500 ppm
I20260619 02:44:13.331555 8788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:13 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837053, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:13.334964 8788 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:13.336457 8788 webserver.cc:460] Webserver started at http://127.7.246.65:32983/ using document root <none> and password file <none>
I20260619 02:44:13.337126 8788 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:13.337239 8788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:13.337495 8788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:13.340590 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "487f5cf56c5e1f49ebc0ed424b9156c3"
server_key_iv: "f91a3065437b0ae96cac7ffba6627814"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.341315 8788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "487f5cf56c5e1f49ebc0ed424b9156c3"
server_key_iv: "f91a3065437b0ae96cac7ffba6627814"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.346705 8788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:13.350221 8803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:13.351573 8788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:13.351744 8788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "487f5cf56c5e1f49ebc0ed424b9156c3"
server_key_iv: "f91a3065437b0ae96cac7ffba6627814"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.351954 8788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:13.366159 8788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:13.370952 8788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:13.371253 8788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:13.372177 8788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:13.373508 8788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:13.373638 8788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 02:44:13.373735 8788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:13.373781 8788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:13.407001 8788 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:34533
I20260619 02:44:13.407043 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:34533 every 8 connection(s)
I20260619 02:44:13.408282 8788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:13.410880 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8788
I20260619 02:44:13.411118 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:13.411962 8153 external_mini_cluster.cc:1428] Setting key 625576df46743563c1eac76861bb7ce9
Jun 19 02:44:13 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837053, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:13.430217 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:13.410868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:41749 (local address 127.7.246.126:46773)
0619 02:44:13.411680 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:13.411686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:13.412538 (+ 852us) server_negotiation.cc:407] Connection header received
0619 02:44:13.414667 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:13.414674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:13.414790 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:13.414908 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:13.416363 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.417521 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.419894 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.420277 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.420379 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:13.424578 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:13.424584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:13.424587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:13.424619 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:13.427257 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:13.428001 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:13.428007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:13.428010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:13.428130 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:13.428547 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:13.428554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:13.428557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:13.428881 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:13.429027 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:13.429789 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 02:44:13.429958 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":587,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:44:13.431470 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:13.411494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:46773 (local address 127.7.246.65:41749)
0619 02:44:13.412364 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:13.412414 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:13.414410 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:13.415101 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:13.415108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:13.415607 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:13.416185 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:13.416202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.417705 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.417709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:13.419738 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:13.419755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.420504 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.420514 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:13.420855 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:13.421640 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:13.421675 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:13.424339 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:13.427409 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:13.427419 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:13.427438 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:13.427864 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:13.428238 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:13.428243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:13.428247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:13.428443 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:13.429004 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:13.429012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:13.429381 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 02:44:13.429712 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 02:44:13.430126 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":316,"threads_started":1}
I20260619 02:44:13.433290 8916 heartbeater.cc:347] Connected to a master server at 127.7.246.126:46773
I20260619 02:44:13.433652 8916 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:13.434414 8916 heartbeater.cc:511] Master 127.7.246.126:46773 requested a full tablet report, sending...
I20260619 02:44:13.436424 8727 ts_manager.cc:194] Registered new tserver with Master: 7e4e024a8a9f405e8ea9fd3cc70b0f82 (127.7.246.65:34533)
I20260619 02:44:13.437877 8727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:41749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:13.453130 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:13.786028 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:13.786319 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:13.786412 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:13.793884 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:13.793993 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:13.794107 8922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:13.804483 8922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:13.805922 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:13.807322 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:13.820297 8928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:13.820420 8927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:13.820588 8930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:13.821131 8922 server_base.cc:1034] running on GCE node
I20260619 02:44:13.821766 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:13.822723 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:13.823958 8922 hybrid_clock.cc:648] HybridClock initialized: now 1781837053823909 us; error 42 us; skew 500 ppm
I20260619 02:44:13.824281 8922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:13 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837053, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:13.827673 8922 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:13.829295 8922 webserver.cc:460] Webserver started at http://127.7.246.66:33067/ using document root <none> and password file <none>
I20260619 02:44:13.830190 8922 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:13.830312 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:13.830605 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:13.834308 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "f891499992f44319a5239a12ebb6a320"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "544d8e9660edcb7239abe4e976beb57c"
server_key_iv: "0e9e54af2b531d87f9f690ee0ea82188"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.835031 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "f891499992f44319a5239a12ebb6a320"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "544d8e9660edcb7239abe4e976beb57c"
server_key_iv: "0e9e54af2b531d87f9f690ee0ea82188"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.840561 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:13.843924 8937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:13.845229 8922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:13.845362 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "f891499992f44319a5239a12ebb6a320"
format_stamp: "Formatted at 2026-06-19 02:44:13 on dist-test-slave-9tkz"
server_key: "544d8e9660edcb7239abe4e976beb57c"
server_key_iv: "0e9e54af2b531d87f9f690ee0ea82188"
server_key_version: "encryptionkey@0"
I20260619 02:44:13.845522 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:13.890573 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:13.895002 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:13.895362 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:13.896370 8922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:13.897832 8922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:13.897946 8922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:13.898113 8922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:13.898202 8922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:13.932824 8922 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:38869
I20260619 02:44:13.932873 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:38869 every 8 connection(s)
I20260619 02:44:13.934214 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:13.941459 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 8922
I20260619 02:44:13.941679 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:13.942242 8153 external_mini_cluster.cc:1428] Setting key 7e67a4bc4ac7e1581381cec35c949f56
Jun 19 02:44:13 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837053, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:13.957134 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:13.936935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:56543 (local address 127.7.246.126:46773)
0619 02:44:13.937385 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:13.937389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:13.938435 (+ 1046us) server_negotiation.cc:407] Connection header received
0619 02:44:13.940205 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:13.940210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:13.940298 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:13.940412 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:13.942288 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.943304 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.945367 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.945878 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.945986 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:13.950306 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:13.950317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:13.950321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:13.950352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:13.953480 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:13.954233 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:13.954239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:13.954242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:13.954363 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:13.954852 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:13.954856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:13.954860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:13.955137 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:13.955294 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:13.956295 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0619 02:44:13.956596 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":267,"threads_started":1}
I20260619 02:44:13.957856 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:13.937619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:46773 (local address 127.7.246.66:56543)
0619 02:44:13.938275 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:13.938318 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:13.939948 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:13.940639 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:13.940647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:13.941255 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:13.942083 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:13.942106 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.943449 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.943453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:13.945217 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:13.945234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:13.946110 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:13.946120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:13.946475 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:13.947236 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:13.947271 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:13.950071 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:13.953655 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:13.953664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:13.953683 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:13.954106 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:13.954478 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:13.954485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:13.954489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:13.954732 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:13.955312 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:13.955322 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:13.955787 (+ 465us) client_negotiation.cc:769] Sending connection context
0619 02:44:13.956119 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 02:44:13.956648 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260619 02:44:13.959391 9050 heartbeater.cc:347] Connected to a master server at 127.7.246.126:46773
I20260619 02:44:13.959805 9050 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:13.960815 9050 heartbeater.cc:511] Master 127.7.246.126:46773 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:44:13.962581 8727 ts_manager.cc:194] Registered new tserver with Master: f891499992f44319a5239a12ebb6a320 (127.7.246.66:38869)
I20260619 02:44:13.963487 8727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:56543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:13.985280 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:14.301880 9056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:14.302137 9056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:14.302326 9056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:14.309334 9056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:14.309464 9056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:14.309677 9056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:14.319142 9056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:46773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:14.320396 9056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:14.321677 9056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:14.334609 9061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:14.334550 9062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:14.334584 9064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:14.335510 9056 server_base.cc:1034] running on GCE node
I20260619 02:44:14.336097 9056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:14.336923 9056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:14.338140 9056 hybrid_clock.cc:648] HybridClock initialized: now 1781837054338081 us; error 52 us; skew 500 ppm
I20260619 02:44:14.338470 9056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:14.341512 9056 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:14.342998 9056 webserver.cc:460] Webserver started at http://127.7.246.67:32841/ using document root <none> and password file <none>
I20260619 02:44:14.343647 9056 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:14.343791 9056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:14.344091 9056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:14.347019 9056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f"
format_stamp: "Formatted at 2026-06-19 02:44:14 on dist-test-slave-9tkz"
server_key: "48789dd48aae1961f82bff147034b7dc"
server_key_iv: "3aa4fe553159db640cbe25187f5581f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:14.347816 9056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f"
format_stamp: "Formatted at 2026-06-19 02:44:14 on dist-test-slave-9tkz"
server_key: "48789dd48aae1961f82bff147034b7dc"
server_key_iv: "3aa4fe553159db640cbe25187f5581f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:14.352957 9056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:44:14.356389 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:14.357795 9056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:44:14.357955 9056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f"
format_stamp: "Formatted at 2026-06-19 02:44:14 on dist-test-slave-9tkz"
server_key: "48789dd48aae1961f82bff147034b7dc"
server_key_iv: "3aa4fe553159db640cbe25187f5581f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:14.358144 9056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:14.383186 9056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:14.387579 9056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:14.387895 9056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:14.388844 9056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:14.390198 9056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:14.390300 9056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:14.390447 9056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:14.390517 9056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:14.422355 9056 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:35911
I20260619 02:44:14.422398 9183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:35911 every 8 connection(s)
I20260619 02:44:14.423607 9056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:14.431749 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9056
I20260619 02:44:14.431965 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:14.432444 8153 external_mini_cluster.cc:1428] Setting key 6252b7fea084334bd201d53e5a1e9df6
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:14.440989 8916 heartbeater.cc:503] Master 127.7.246.126:46773 was elected leader, sending a full tablet report...
I20260619 02:44:14.444092 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.426110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:34009 (local address 127.7.246.126:46773)
0619 02:44:14.426284 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.426288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.427354 (+ 1066us) server_negotiation.cc:407] Connection header received
0619 02:44:14.429115 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.429121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.429211 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.429329 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.430817 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.431514 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.433464 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.434231 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.434362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.438343 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.438361 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.438364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.438397 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.441092 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.441885 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.441889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.441891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.442023 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.442523 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.442529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.442532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.442932 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.443057 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.443681 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.443847 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:44:14.445019 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.426567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:46773 (local address 127.7.246.67:34009)
0619 02:44:14.427203 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.427246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.428842 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.429602 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.429616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.430109 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:14.430654 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.430670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.431685 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.431695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.433102 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.433117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.434463 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.434475 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.434754 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.435440 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:14.435468 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:14.438139 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:14.441254 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:14.441260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:14.441274 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:14.441757 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:14.442132 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:14.442137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:14.442140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:14.442374 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:14.443050 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:14.443057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:14.443359 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.443588 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.443949 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":246,"threads_started":1}
I20260619 02:44:14.446609 9184 heartbeater.cc:347] Connected to a master server at 127.7.246.126:46773
I20260619 02:44:14.446954 9184 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:14.447760 9184 heartbeater.cc:511] Master 127.7.246.126:46773 requested a full tablet report, sending...
I20260619 02:44:14.449198 8727 ts_manager.cc:194] Registered new tserver with Master: 8fd95cda3a2a44d9b0dc0b30b54fee0f (127.7.246.67:35911)
I20260619 02:44:14.450151 8727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:34009
I20260619 02:44:14.460302 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:14.479058 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.466516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.7.246.126:46773)
0619 02:44:14.466714 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.466719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.467175 (+ 456us) server_negotiation.cc:407] Connection header received
0619 02:44:14.467391 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.467396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.467470 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.467573 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.468946 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.469762 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.471447 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.471861 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.471991 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.474413 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.474418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.474421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.474454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.476870 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.477438 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.477442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.477444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.477619 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.477923 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.477926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.477928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.478196 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.478386 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.478659 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.478803 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:44:14.485316 8727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:14.492189 8727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:14.514452 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.504642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:35911 (local address 127.0.0.1:44026)
0619 02:44:14.505472 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.505490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.505662 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.506851 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.506858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.506884 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:14.507318 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.507333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.509896 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.509902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.512035 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.512044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.513315 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.513322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.513516 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.513632 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.513763 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.513955 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":71,"thread_start_us":1682,"threads_started":1}
I20260619 02:44:14.514864 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.505328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.7.246.67:35911)
0619 02:44:14.506248 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.506256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.506284 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:44:14.506386 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.506393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.506631 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.507924 (+ 1293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:14.508196 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.509769 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.512172 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.513125 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.513253 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.513472 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.514240 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.514422 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":617,"threads_started":1}
I20260619 02:44:14.515139 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.504375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.7.246.65:34533)
0619 02:44:14.504973 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.504984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.505519 (+ 535us) server_negotiation.cc:407] Connection header received
0619 02:44:14.505643 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.505653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.505910 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.506100 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:14.506947 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.509063 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.511736 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.513181 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.513335 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.513561 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.514650 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.514813 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":296,"threads_started":1}
I20260619 02:44:14.515509 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.504237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:34533 (local address 127.0.0.1:57450)
0619 02:44:14.505047 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.505089 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.505289 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.506236 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.506244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.506277 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:14.506748 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.506764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.509207 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.509212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.511540 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.511559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.513674 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.513680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.513877 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.513918 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.513998 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.514128 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":751,"threads_started":1}
I20260619 02:44:14.516079 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.506232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.7.246.66:38869)
0619 02:44:14.508343 (+ 2111us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.508350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.508379 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:44:14.508445 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.508450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.508597 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.508770 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:14.509271 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.510701 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.513085 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.514545 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.514673 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.514855 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.515602 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.515778 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1969,"thread_start_us":1870,"threads_started":1}
I20260619 02:44:14.516284 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.505785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:38869 (local address 127.0.0.1:50904)
0619 02:44:14.506956 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.506972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.507082 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.508759 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.508764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.508789 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:14.509099 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.509107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.510866 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.510875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.512916 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.512931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.514973 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.514979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.515140 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.515170 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.515264 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.515402 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":25,"thread_start_us":878,"threads_started":1}
I20260619 02:44:14.519187 8851 tablet_service.cc:1467] Processing CreateTablet for tablet 697a57edb1834e77b28f626868ac467b (DEFAULT_TABLE table=test-table [id=a315546c138545d59f047f8f0e7d90fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:14.519224 9119 tablet_service.cc:1467] Processing CreateTablet for tablet 697a57edb1834e77b28f626868ac467b (DEFAULT_TABLE table=test-table [id=a315546c138545d59f047f8f0e7d90fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:14.519840 8985 tablet_service.cc:1467] Processing CreateTablet for tablet 697a57edb1834e77b28f626868ac467b (DEFAULT_TABLE table=test-table [id=a315546c138545d59f047f8f0e7d90fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:14.520471 8851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a57edb1834e77b28f626868ac467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:14.520614 9119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a57edb1834e77b28f626868ac467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:14.520978 8985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697a57edb1834e77b28f626868ac467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:14.531949 9202 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Bootstrap starting.
I20260619 02:44:14.535323 9202 tablet_bootstrap.cc:654] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:14.535882 9203 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Bootstrap starting.
I20260619 02:44:14.538595 9202 log.cc:824] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:14.539379 9203 tablet_bootstrap.cc:654] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:14.540045 9204 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Bootstrap starting.
I20260619 02:44:14.540607 9203 log.cc:824] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:14.542160 9202 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: No bootstrap required, opened a new log
I20260619 02:44:14.542425 9202 ts_tablet_manager.cc:1397] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 02:44:14.543380 9204 tablet_bootstrap.cc:654] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:14.543635 9203 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: No bootstrap required, opened a new log
I20260619 02:44:14.543907 9203 ts_tablet_manager.cc:1397] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:44:14.544566 9204 log.cc:824] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:14.547821 9204 tablet_bootstrap.cc:492] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: No bootstrap required, opened a new log
I20260619 02:44:14.548069 9204 ts_tablet_manager.cc:1397] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 02:44:14.551299 9202 raft_consensus.cc:348] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.551647 9202 raft_consensus.cc:374] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:14.551729 9202 raft_consensus.cc:729] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e4e024a8a9f405e8ea9fd3cc70b0f82, State: Initialized, Role: FOLLOWER
I20260619 02:44:14.552299 9202 consensus_queue.cc:260] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.553951 9202 ts_tablet_manager.cc:1428] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 02:44:14.554136 9203 raft_consensus.cc:348] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.554549 9203 raft_consensus.cc:374] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:14.554649 9203 raft_consensus.cc:729] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd95cda3a2a44d9b0dc0b30b54fee0f, State: Initialized, Role: FOLLOWER
I20260619 02:44:14.555327 9203 consensus_queue.cc:260] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.556447 9184 heartbeater.cc:503] Master 127.7.246.126:46773 was elected leader, sending a full tablet report...
I20260619 02:44:14.558434 9203 ts_tablet_manager.cc:1428] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 02:44:14.559698 9204 raft_consensus.cc:348] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.560156 9204 raft_consensus.cc:374] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:14.560274 9204 raft_consensus.cc:729] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f891499992f44319a5239a12ebb6a320, State: Initialized, Role: FOLLOWER
I20260619 02:44:14.560746 9204 consensus_queue.cc:260] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.561782 9050 heartbeater.cc:503] Master 127.7.246.126:46773 was elected leader, sending a full tablet report...
I20260619 02:44:14.562491 9204 ts_tablet_manager.cc:1428] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 02:44:14.595888 9210 raft_consensus.cc:482] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:14.596130 9210 raft_consensus.cc:504] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.597694 9210 leader_election.cc:288] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e4e024a8a9f405e8ea9fd3cc70b0f82 (127.7.246.65:34533), 8fd95cda3a2a44d9b0dc0b30b54fee0f (127.7.246.67:35911)
I20260619 02:44:14.605360 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.598235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:32995 (local address 127.7.246.67:35911)
0619 02:44:14.598473 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.598482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.599379 (+ 897us) server_negotiation.cc:407] Connection header received
0619 02:44:14.599568 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.599574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.599640 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.599737 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:14.600386 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.601584 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.603720 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.604470 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.604653 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.604725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.604994 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.605111 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:44:14.605458 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.598152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:53511 (local address 127.7.246.65:34533)
0619 02:44:14.598422 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.598430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.599258 (+ 828us) server_negotiation.cc:407] Connection header received
0619 02:44:14.599477 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.599483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.599554 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.599649 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:14.600261 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.601353 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.603791 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.604559 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.604732 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.604805 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.605075 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.605206 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:44:14.605844 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.598213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:35911 (local address 127.7.246.66:32995)
0619 02:44:14.599279 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.599293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.599432 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.599751 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.599755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.599791 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:14.600248 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.600256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.601702 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.601707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.603574 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.603585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.604681 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.604687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.604849 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.604871 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.604973 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.605089 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":21,"thread_start_us":444,"threads_started":1}
I20260619 02:44:14.605901 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.598083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:34533 (local address 127.7.246.66:53511)
0619 02:44:14.599146 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:14.599167 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:14.599337 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:14.599668 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:14.599673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:14.599694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:14.600089 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.600097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.601505 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.601510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.603612 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:14.603623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.604774 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.604780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:14.604930 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.604951 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:44:14.605052 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:44:14.605165 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":45,"thread_start_us":671,"threads_started":1}
I20260619 02:44:14.606843 8871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a57edb1834e77b28f626868ac467b" candidate_uuid: "f891499992f44319a5239a12ebb6a320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" is_pre_election: true
I20260619 02:44:14.606833 9139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a57edb1834e77b28f626868ac467b" candidate_uuid: "f891499992f44319a5239a12ebb6a320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" is_pre_election: true
I20260619 02:44:14.607342 9139 raft_consensus.cc:2454] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f891499992f44319a5239a12ebb6a320 in term 0.
I20260619 02:44:14.607388 8871 raft_consensus.cc:2454] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f891499992f44319a5239a12ebb6a320 in term 0.
I20260619 02:44:14.608042 8939 leader_election.cc:302] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e4e024a8a9f405e8ea9fd3cc70b0f82, f891499992f44319a5239a12ebb6a320; no voters:
I20260619 02:44:14.608467 9210 raft_consensus.cc:2781] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:14.608597 9210 raft_consensus.cc:482] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:14.608670 9210 raft_consensus.cc:3037] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:14.611011 9210 raft_consensus.cc:504] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.611503 9210 leader_election.cc:288] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [CANDIDATE]: Term 1 election: Requested vote from peers 7e4e024a8a9f405e8ea9fd3cc70b0f82 (127.7.246.65:34533), 8fd95cda3a2a44d9b0dc0b30b54fee0f (127.7.246.67:35911)
I20260619 02:44:14.611966 8871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a57edb1834e77b28f626868ac467b" candidate_uuid: "f891499992f44319a5239a12ebb6a320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82"
I20260619 02:44:14.612006 9139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "697a57edb1834e77b28f626868ac467b" candidate_uuid: "f891499992f44319a5239a12ebb6a320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f"
I20260619 02:44:14.612149 8871 raft_consensus.cc:3037] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:14.612286 9139 raft_consensus.cc:3037] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:14.614382 8871 raft_consensus.cc:2454] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f891499992f44319a5239a12ebb6a320 in term 1.
I20260619 02:44:14.614753 8939 leader_election.cc:302] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e4e024a8a9f405e8ea9fd3cc70b0f82, f891499992f44319a5239a12ebb6a320; no voters:
I20260619 02:44:14.615132 9210 raft_consensus.cc:2781] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:14.615624 9139 raft_consensus.cc:2454] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f891499992f44319a5239a12ebb6a320 in term 1.
I20260619 02:44:14.615643 9210 raft_consensus.cc:686] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [term 1 LEADER]: Becoming Leader. State: Replica: f891499992f44319a5239a12ebb6a320, State: Running, Role: LEADER
I20260619 02:44:14.616216 9210 consensus_queue.cc:237] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } }
I20260619 02:44:14.620935 8726 catalog_manager.cc:5495] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320 reported cstate change: term changed from 0 to 1, leader changed from <none> to f891499992f44319a5239a12ebb6a320 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "f891499992f44319a5239a12ebb6a320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e4e024a8a9f405e8ea9fd3cc70b0f82" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f891499992f44319a5239a12ebb6a320" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd95cda3a2a44d9b0dc0b30b54fee0f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35911 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:14.650956 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.635718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.7.246.65:34533)
0619 02:44:14.635881 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.635886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.636380 (+ 494us) server_negotiation.cc:407] Connection header received
0619 02:44:14.636537 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.636540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.636604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.636682 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.638647 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.639348 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.640544 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.641098 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.641227 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.645893 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.645906 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.645921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.645968 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.648574 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.649174 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.649181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.649183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.649348 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.649695 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.649698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.649700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.650029 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.650298 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.650494 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.650698 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:44:14.660576 8917 tablet.cc:2369] T 697a57edb1834e77b28f626868ac467b P 7e4e024a8a9f405e8ea9fd3cc70b0f82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:44:14.675221 9185 tablet.cc:2369] T 697a57edb1834e77b28f626868ac467b P 8fd95cda3a2a44d9b0dc0b30b54fee0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 02:44:14.686350 9051 tablet.cc:2369] T 697a57edb1834e77b28f626868ac467b P f891499992f44319a5239a12ebb6a320: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:14.697952 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.682776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.7.246.65:34533)
0619 02:44:14.682999 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.683009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.683305 (+ 296us) server_negotiation.cc:407] Connection header received
0619 02:44:14.683516 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.683520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.683584 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.683691 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.685015 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.685822 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.687952 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.688363 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.688498 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.692803 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.692809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.692818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.692858 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.695416 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.696011 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.696018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.696022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.696172 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.696590 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.696596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.696599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.696853 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.697123 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.697372 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.697618 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:14.741271 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.727310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.7.246.65:34533)
0619 02:44:14.727567 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.727576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.727815 (+ 239us) server_negotiation.cc:407] Connection header received
0619 02:44:14.728025 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.728029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.728092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.728206 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.729625 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.730289 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.731443 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.731855 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.731978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.736082 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.736086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.736090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.736122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.738686 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.739329 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.739336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.739339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.739460 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.739840 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.739846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.739849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.740142 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.740394 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.740682 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.740915 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 02:44:14.743045 8831 tablet_service.cc:2908] Rejecting scan request for tablet 697a57edb1834e77b28f626868ac467b: Uninitialized: clean time has not yet been initialized
I20260619 02:44:14.745949 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8788
I20260619 02:44:14.758262 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.746877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.7.246.65:34533)
0619 02:44:14.747017 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.747021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.747424 (+ 403us) server_negotiation.cc:407] Connection header received
0619 02:44:14.747538 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.747541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.747595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.747689 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.748777 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.749438 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.750606 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.751013 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.751110 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.753103 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.753107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.753110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.753135 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.755606 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.756293 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.756297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.756299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.756409 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.756790 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.756793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.756795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.757042 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.757264 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.757634 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.757967 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:44:14.758899 8911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:14.845605 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8788
I20260619 02:44:14.863443 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8922
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:14.877915 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.864186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.7.246.66:38869)
0619 02:44:14.864433 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.864439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.864458 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:44:14.864557 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.864562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.864633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.864736 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.865973 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.866752 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.867983 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.868383 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.868512 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.872350 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.872372 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.872387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.872426 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:14.875266 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.875930 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.875937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.875940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.876083 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:14.876435 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:14.876442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:14.876445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:14.876763 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:14.877061 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:14.877333 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 02:44:14.877605 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 02:44:14.878670 9045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:14.970211 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8922
I20260619 02:44:14.990464 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9056
Jun 19 02:44:14 dist-test-slave-9tkz krb5kdc[8682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:15.004738 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:14.991213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.7.246.67:35911)
0619 02:44:14.991405 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:14.991412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:14.991431 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:44:14.991598 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:14.991606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:14.991680 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:14.991783 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:14.992969 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.993811 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.995062 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:14.995476 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:14.995602 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:14.999560 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:14.999573 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:14.999588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:14.999618 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:15.002205 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:15.002835 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:15.002844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:15.002849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:15.002971 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:15.003330 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:15.003335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:15.003344 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:15.003663 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:15.003931 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:15.004209 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:44:15.004447 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:44:15.005353 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:15.092478 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9056
I20260619 02:44:15.108852 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 8695
I20260619 02:44:15.110204 8757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:15.214169 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 8695
2026-06-19T02:44:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4079 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:15 dist-test-slave-9tkz krb5kdc[9232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:15 dist-test-slave-9tkz krb5kdc[9232](info): set up 2 sockets
Jun 19 02:44:15 dist-test-slave-9tkz krb5kdc[9232](info): commencing operation
krb5kdc: starting...
W20260619 02:44:16.297840 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:16 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:16.482277 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43963
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:16.810688 9245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:16.810979 9245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:16.811151 9245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:16.818361 9245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:16.818459 9245 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:44:16.818547 9245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:16.818651 9245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:16.818743 9245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:16.818835 9245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:16.828584 9245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:16.829885 9245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:16.831143 9245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:16.843254 9250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:16.843313 9251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:16.843282 9253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:16.844094 9245 server_base.cc:1034] running on GCE node
I20260619 02:44:16.844816 9245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:16.846238 9245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:16.847457 9245 hybrid_clock.cc:648] HybridClock initialized: now 1781837056847393 us; error 36 us; skew 500 ppm
I20260619 02:44:16.847807 9245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:16 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837056, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:16.850950 9245 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:16.852401 9245 webserver.cc:460] Webserver started at http://127.7.246.126:36251/ using document root <none> and password file <none>
I20260619 02:44:16.853106 9245 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:16.853235 9245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:16.853598 9245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:16.856493 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "f92b2897b2a7445bbbc86b548581321b"
format_stamp: "Formatted at 2026-06-19 02:44:16 on dist-test-slave-9tkz"
server_key: "d3e541e82faf4844191afdcf973e2e18"
server_key_iv: "225c571528b706a0c9b2b292c23f277f"
server_key_version: "encryptionkey@0"
I20260619 02:44:16.857277 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "f92b2897b2a7445bbbc86b548581321b"
format_stamp: "Formatted at 2026-06-19 02:44:16 on dist-test-slave-9tkz"
server_key: "d3e541e82faf4844191afdcf973e2e18"
server_key_iv: "225c571528b706a0c9b2b292c23f277f"
server_key_version: "encryptionkey@0"
I20260619 02:44:16.862357 9245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 02:44:16.865868 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:16.867208 9245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:44:16.867391 9245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "f92b2897b2a7445bbbc86b548581321b"
format_stamp: "Formatted at 2026-06-19 02:44:16 on dist-test-slave-9tkz"
server_key: "d3e541e82faf4844191afdcf973e2e18"
server_key_iv: "225c571528b706a0c9b2b292c23f277f"
server_key_version: "encryptionkey@0"
I20260619 02:44:16.867616 9245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:16.887976 9245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:16.892409 9245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:16.892686 9245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:16.914397 9245 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43963
I20260619 02:44:16.914417 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43963 every 8 connection(s)
I20260619 02:44:16.915777 9245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:16.919555 9322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:16.919849 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9245
I20260619 02:44:16.920063 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:16.920893 8153 external_mini_cluster.cc:1428] Setting key f9cf6bc20585626e3330d7e5bd140432
I20260619 02:44:16.933698 9322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Bootstrap starting.
Jun 19 02:44:16 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:16.937470 9322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:16.938558 9322 log.cc:824] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:16.941265 9322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: No bootstrap required, opened a new log
I20260619 02:44:16.944591 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:16.923108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.7.246.126:43963)
0619 02:44:16.924010 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:16.924022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:16.924066 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:44:16.926021 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:16.926045 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:16.926489 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:16.926869 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:16.928516 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:16.929978 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:16.931783 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:16.932469 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:16.932629 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:16.937408 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:16.937416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:16.937430 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:16.937469 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:16.940638 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:16.941313 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:16.941326 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:16.941331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:16.941589 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:16.941960 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:16.941966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:16.941970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:16.942453 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:16.942769 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:16.943146 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 02:44:16.943571 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":459,"threads_started":1}
I20260619 02:44:16.948954 9322 raft_consensus.cc:348] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } }
I20260619 02:44:16.949316 9322 raft_consensus.cc:374] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:16.949415 9322 raft_consensus.cc:729] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92b2897b2a7445bbbc86b548581321b, State: Initialized, Role: FOLLOWER
I20260619 02:44:16.949956 9322 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } }
I20260619 02:44:16.950191 9322 raft_consensus.cc:388] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:16.950327 9322 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:16.950489 9322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:16.952687 9322 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } }
I20260619 02:44:16.953166 9322 leader_election.cc:302] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f92b2897b2a7445bbbc86b548581321b; no voters:
I20260619 02:44:16.953840 9322 leader_election.cc:288] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:16.953971 9327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:16.954685 9327 raft_consensus.cc:686] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [term 1 LEADER]: Becoming Leader. State: Replica: f92b2897b2a7445bbbc86b548581321b, State: Running, Role: LEADER
I20260619 02:44:16.955271 9327 consensus_queue.cc:237] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } }
I20260619 02:44:16.955982 9322 sys_catalog.cc:564] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:16.958091 9329 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92b2897b2a7445bbbc86b548581321b. Latest consensus state: current_term: 1 leader_uuid: "f92b2897b2a7445bbbc86b548581321b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } } }
I20260619 02:44:16.958101 9328 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f92b2897b2a7445bbbc86b548581321b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f92b2897b2a7445bbbc86b548581321b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43963 } } }
I20260619 02:44:16.958647 9329 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:16.958647 9328 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:16.966254 9333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:16.970973 9333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:16.978955 9333 catalog_manager.cc:1347] Generated new cluster ID: 17f8fb247f4646e4b2a8403954b69d7c
I20260619 02:44:16.979076 9333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:17.012343 9333 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:17.013959 9333 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:17.023105 9333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Generated new TSK 0
I20260619 02:44:17.024170 9333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:17.080535 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43963
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:17.406038 9348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:17.406314 9348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:17.406498 9348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:17.413929 9348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:17.414104 9348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:17.414268 9348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:17.414397 9348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:17.424119 9348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:43963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:17.425393 9348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:17.426708 9348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:17.440256 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:17.440346 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:17.440384 9356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:17.440975 9348 server_base.cc:1034] running on GCE node
I20260619 02:44:17.441705 9348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:17.442689 9348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:17.443919 9348 hybrid_clock.cc:648] HybridClock initialized: now 1781837057443840 us; error 59 us; skew 500 ppm
I20260619 02:44:17.444209 9348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:17 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:17.447417 9348 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:17.448865 9348 webserver.cc:460] Webserver started at http://127.7.246.65:46003/ using document root <none> and password file <none>
I20260619 02:44:17.449658 9348 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:17.449784 9348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:17.450089 9348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:17.452873 9348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "1b27f767b8ea7633d681666242f3fa10"
server_key_iv: "c0f0946d878a923819b51d1ae8765a52"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.453752 9348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "1b27f767b8ea7633d681666242f3fa10"
server_key_iv: "c0f0946d878a923819b51d1ae8765a52"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.458952 9348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 02:44:17.462281 9363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:17.463505 9348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:17.463672 9348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "1b27f767b8ea7633d681666242f3fa10"
server_key_iv: "c0f0946d878a923819b51d1ae8765a52"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.463891 9348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:17.490679 9348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:17.494946 9348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:17.495231 9348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:17.497885 9348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:17.497991 9348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:17.498158 9348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:17.498234 9348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:17 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:17.523015 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:17.498803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:53627 (local address 127.7.246.126:43963)
0619 02:44:17.499268 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:17.499274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:17.500627 (+ 1353us) server_negotiation.cc:407] Connection header received
0619 02:44:17.502982 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:17.502991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:17.503098 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:17.503243 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:17.505689 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.506973 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.508729 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.509505 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.509671 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:17.514972 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:17.514982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:17.514990 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:17.515043 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:17.519066 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:17.520106 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:17.520110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:17.520112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:17.520224 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:17.520829 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:17.520834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:17.520838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:17.521270 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:17.521430 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:17.522417 (+ 987us) server_negotiation.cc:299] Negotiation successful
0619 02:44:17.522625 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260619 02:44:17.524084 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:17.499493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.65:53627)
0619 02:44:17.500439 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:17.500484 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:17.502550 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:17.503538 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:17.503546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:17.504400 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:17.505426 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:17.505484 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.507209 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.507213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:17.508583 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:17.508598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.509826 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.509836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:17.510158 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:17.510974 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:17.511008 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:17.514712 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:17.519257 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:17.519268 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:17.519293 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:17.519926 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:17.520390 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:17.520397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:17.520403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:17.520698 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:17.521402 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:17.521414 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:17.521904 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 02:44:17.522323 (+ 419us) client_negotiation.cc:240] Negotiation successful
0619 02:44:17.522893 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":395,"threads_started":1}
I20260619 02:44:17.544642 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:17.530107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:54399 (local address 127.7.246.126:43963)
0619 02:44:17.530293 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:17.530302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:17.531014 (+ 712us) server_negotiation.cc:407] Connection header received
0619 02:44:17.531258 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:17.531264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:17.531350 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:17.531467 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:17.532705 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.533799 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.535809 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.536341 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.536499 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:17.538631 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:17.538639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:17.538646 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:17.538691 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:17.541782 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:17.542776 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:17.542785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:17.542790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:17.542917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:17.543282 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:17.543287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:17.543290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:17.543541 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:17.543687 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:17.544087 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:44:17.544303 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:44:17.544750 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:17.530173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.65:54399)
0619 02:44:17.530881 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:17.530905 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:17.531089 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:17.531502 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:17.531508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:17.532206 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:17.532563 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:17.532572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.533955 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.533960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:17.535654 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:17.535665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:17.536760 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:17.536767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:17.536900 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:17.537721 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:17.537738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:17.538450 (+ 712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:17.541962 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:17.541971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:17.541975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:17.542651 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:17.543023 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:17.543029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:17.543033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:17.543166 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:17.543654 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:17.543659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:17.543842 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 02:44:17.543982 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:44:17.544313 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":341,"threads_started":1}
I20260619 02:44:17.552423 9348 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33973
I20260619 02:44:17.552457 9483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33973 every 8 connection(s)
I20260619 02:44:17.553594 9348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:17.555531 9484 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43963
I20260619 02:44:17.555766 9484 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:17.556442 9484 heartbeater.cc:511] Master 127.7.246.126:43963 requested a full tablet report, sending...
I20260619 02:44:17.558425 9277 ts_manager.cc:194] Registered new tserver with Master: 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973)
I20260619 02:44:17.559259 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9348
I20260619 02:44:17.559466 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:17.559968 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:53627
I20260619 02:44:17.560104 8153 external_mini_cluster.cc:1428] Setting key 310ddd4d92c05c19fcab4c4868d9d03a
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:17.601691 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43963
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:17.915416 9488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:17.915726 9488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:17.915911 9488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:17.923629 9488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:17.923784 9488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:17.923924 9488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:17.924059 9488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:17.933648 9488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:43963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:17.934873 9488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:17.936254 9488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:17.949620 9494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:17.949620 9496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:17.949837 9493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:17.950552 9488 server_base.cc:1034] running on GCE node
I20260619 02:44:17.951220 9488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:17.952153 9488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:17.953365 9488 hybrid_clock.cc:648] HybridClock initialized: now 1781837057953301 us; error 38 us; skew 500 ppm
I20260619 02:44:17.953719 9488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:17 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:17.960209 9488 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:17.961779 9488 webserver.cc:460] Webserver started at http://127.7.246.66:41307/ using document root <none> and password file <none>
I20260619 02:44:17.962484 9488 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:17.962608 9488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:17.962926 9488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:17.965909 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "7bd4d02899de4875a3f594fc5d50df5c"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "065aeabe316aa7df3f08e78df12942ff"
server_key_iv: "9084c19ee4cc15a01e18140de4684b9e"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.966809 9488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd4d02899de4875a3f594fc5d50df5c"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "065aeabe316aa7df3f08e78df12942ff"
server_key_iv: "9084c19ee4cc15a01e18140de4684b9e"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.972033 9488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 02:44:17.975484 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:17.976848 9488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:17.977033 9488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "7bd4d02899de4875a3f594fc5d50df5c"
format_stamp: "Formatted at 2026-06-19 02:44:17 on dist-test-slave-9tkz"
server_key: "065aeabe316aa7df3f08e78df12942ff"
server_key_iv: "9084c19ee4cc15a01e18140de4684b9e"
server_key_version: "encryptionkey@0"
I20260619 02:44:17.977260 9488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:18.009085 9488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:18.013758 9488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:18.014038 9488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:18.016770 9488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:18.016877 9488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:18.017041 9488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:18.017114 9488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:18 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837057, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:18.043408 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.017475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:36631 (local address 127.7.246.126:43963)
0619 02:44:18.017677 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.017684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.019356 (+ 1672us) server_negotiation.cc:407] Connection header received
0619 02:44:18.022721 (+ 3365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.022729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.022848 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.022958 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:18.025165 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.025975 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.029043 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.029860 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.030011 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.035461 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:18.035468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:18.035472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:18.035514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:18.039360 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.040258 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.040265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.040271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.040437 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.040996 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.041002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.041008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.041287 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:18.041425 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.042596 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.042871 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 02:44:18.045146 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.018208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.66:36631)
0619 02:44:18.019134 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.019198 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.022272 (+ 3074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.023333 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.023348 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.024184 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:18.024978 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.024999 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.026139 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.026144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.028866 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.028888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.030209 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.030222 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.030581 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.031496 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:18.031524 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:18.035146 (+ 3622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:18.039503 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.039510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.039523 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.040116 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.040589 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.040599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.040603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.040869 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.041417 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:18.041424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:18.042012 (+ 588us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.042480 (+ 468us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.043148 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":332,"threads_started":1}
I20260619 02:44:18.059346 9488 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:38867
I20260619 02:44:18.059366 9622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:38867 every 8 connection(s)
I20260619 02:44:18.060997 9488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:18.063261 9623 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43963
I20260619 02:44:18.063513 9623 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:18.064203 9623 heartbeater.cc:511] Master 127.7.246.126:43963 requested a full tablet report, sending...
I20260619 02:44:18.065675 9277 ts_manager.cc:194] Registered new tserver with Master: 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867)
I20260619 02:44:18.066761 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:36631
I20260619 02:44:18.067131 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.052663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:33501 (local address 127.7.246.126:43963)
0619 02:44:18.052903 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.052912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.053633 (+ 721us) server_negotiation.cc:407] Connection header received
0619 02:44:18.053936 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.053942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.054025 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.054126 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:18.055589 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.056639 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.058320 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.058860 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.059004 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.061180 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:18.061185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:18.061188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:18.061217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:18.064325 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.064963 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.064969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.064973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.065123 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.065505 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.065509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.065511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.065999 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:18.066112 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.066511 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.066751 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:44:18.067214 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.052820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.66:33501)
0619 02:44:18.053462 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.053487 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.053723 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.054151 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.054156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.054815 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:18.055357 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.055368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.056857 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.056864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.058172 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.058184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.059050 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.059055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.059252 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.059882 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:18.059904 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:18.060965 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:18.064462 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.064468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.064472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.064850 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.065241 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.065246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.065249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.065400 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.066097 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:18.066101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:18.066277 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.066424 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.066744 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":325,"threads_started":1}
I20260619 02:44:18.068404 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9488
I20260619 02:44:18.068593 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:18.069020 8153 external_mini_cluster.cc:1428] Setting key 2c70c0941b408df51522cda7db0368d5
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:18.106808 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43963
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:18.421278 9627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:18.421658 9627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:18.421850 9627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:18.429411 9627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:18.429533 9627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:18.429759 9627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:18.429895 9627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:18.439769 9627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:43963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:18.441067 9627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:18.442468 9627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:18.455794 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:18.455904 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:18.455989 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:18.462288 9627 server_base.cc:1034] running on GCE node
I20260619 02:44:18.463060 9627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:18.463963 9627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:18.465196 9627 hybrid_clock.cc:648] HybridClock initialized: now 1781837058465125 us; error 38 us; skew 500 ppm
I20260619 02:44:18.465520 9627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:18 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:18.468819 9627 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:18.470305 9627 webserver.cc:460] Webserver started at http://127.7.246.67:40783/ using document root <none> and password file <none>
I20260619 02:44:18.471048 9627 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:18.471182 9627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:18.471536 9627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:18.474407 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "b3fd9e9d1590421b95d926a302f2cdaf"
format_stamp: "Formatted at 2026-06-19 02:44:18 on dist-test-slave-9tkz"
server_key: "f8293d754d3a59de668cc92fd5b4b9a9"
server_key_iv: "d9b71dd1c1c6d6c27a1931df259eee1e"
server_key_version: "encryptionkey@0"
I20260619 02:44:18.475200 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "b3fd9e9d1590421b95d926a302f2cdaf"
format_stamp: "Formatted at 2026-06-19 02:44:18 on dist-test-slave-9tkz"
server_key: "f8293d754d3a59de668cc92fd5b4b9a9"
server_key_iv: "d9b71dd1c1c6d6c27a1931df259eee1e"
server_key_version: "encryptionkey@0"
I20260619 02:44:18.480113 9627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:18.483686 9642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:18.484905 9627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:18.485080 9627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "b3fd9e9d1590421b95d926a302f2cdaf"
format_stamp: "Formatted at 2026-06-19 02:44:18 on dist-test-slave-9tkz"
server_key: "f8293d754d3a59de668cc92fd5b4b9a9"
server_key_iv: "d9b71dd1c1c6d6c27a1931df259eee1e"
server_key_version: "encryptionkey@0"
I20260619 02:44:18.485316 9627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:18.503401 9627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:18.507933 9627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:18.508224 9627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:18.511168 9627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:18.511286 9627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:18.511448 9627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:18.511514 9627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:18 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:18.534495 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.511581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:39443 (local address 127.7.246.126:43963)
0619 02:44:18.511798 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.511806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.513259 (+ 1453us) server_negotiation.cc:407] Connection header received
0619 02:44:18.515424 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.515429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.515522 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.515631 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:18.517913 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.518954 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.521454 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.522143 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.522317 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.526804 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:18.526814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:18.526822 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:18.526877 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:18.530950 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.531891 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.531897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.531900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.532027 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.532595 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.532601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.532605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.532914 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:18.533077 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.533913 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.534136 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:44:18.535535 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.512300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.67:39443)
0619 02:44:18.513093 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.513130 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.515097 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.515904 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.515920 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.516759 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:18.517737 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.517757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.519098 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.519107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.521282 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.521301 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.522419 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.522428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.522693 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.523551 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:18.523586 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:18.526523 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:18.531173 (+ 4650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.531186 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.531210 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.531753 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.532188 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.532196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.532201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.532474 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.533052 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:18.533060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:18.533444 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.533772 (+ 328us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.534349 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":344,"threads_started":1}
I20260619 02:44:18.557686 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.544114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:37355 (local address 127.7.246.126:43963)
0619 02:44:18.544398 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.544403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.544940 (+ 537us) server_negotiation.cc:407] Connection header received
0619 02:44:18.545124 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.545128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.545183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.545256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:18.547263 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.548254 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.549710 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.550299 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.550441 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.552210 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:18.552215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:18.552218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:18.552244 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:18.554998 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.555657 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.555664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.555667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.555832 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.556205 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.556211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.556215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.556539 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:18.556681 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.557076 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.557326 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260619 02:44:18.557745 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.544126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.7.246.67:37355)
0619 02:44:18.544818 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.544842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.545010 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.545322 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.545329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.546129 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:18.546980 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.547000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.548391 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.548397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.549512 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.549523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.550489 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.550495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.550653 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.551267 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:18.551291 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:18.552013 (+ 722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:18.555119 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.555125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.555129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.555547 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.555973 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:18.555977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:18.555981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:18.556114 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:18.556645 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:18.556650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:18.556826 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.556958 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.557294 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":325,"threads_started":1}
I20260619 02:44:18.562996 9484 heartbeater.cc:503] Master 127.7.246.126:43963 was elected leader, sending a full tablet report...
I20260619 02:44:18.576042 9627 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:33859
I20260619 02:44:18.576062 9761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:33859 every 8 connection(s)
I20260619 02:44:18.577291 9627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:18.579329 9762 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43963
I20260619 02:44:18.579542 9762 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:18.580152 9762 heartbeater.cc:511] Master 127.7.246.126:43963 requested a full tablet report, sending...
I20260619 02:44:18.581502 9277 ts_manager.cc:194] Registered new tserver with Master: b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859)
I20260619 02:44:18.582413 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:39443
I20260619 02:44:18.583902 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9627
I20260619 02:44:18.584091 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:18.584580 8153 external_mini_cluster.cc:1428] Setting key d203175f671073f44ca6e305ff9e9383
I20260619 02:44:18.588096 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:44:18 dist-test-slave-9tkz krb5kdc[9232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:44:18 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:18.632756 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.619986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.7.246.126:43963)
0619 02:44:18.620144 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.620150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.620479 (+ 329us) server_negotiation.cc:407] Connection header received
0619 02:44:18.620651 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.620655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.620717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.620812 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:18.622135 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.622849 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.624058 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.624452 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.624535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.628232 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:18.628237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:18.628240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:18.628272 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:18.630694 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.631231 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.631235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.631237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.631344 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:18.631678 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:18.631681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:18.631683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:18.631927 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:18.632112 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.632335 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.632486 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:44:18.636433 9277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:18.643270 9277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:18.667955 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.655084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.7.246.65:33973)
0619 02:44:18.660085 (+ 5001us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.660091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.660113 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:44:18.660186 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.660191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.660374 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.660568 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.661166 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.662671 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.665168 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.666765 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.666935 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.667194 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.667453 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.667626 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4885,"thread_start_us":4776,"threads_started":1}
I20260619 02:44:18.668259 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.655008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33973 (local address 127.0.0.1:53590)
0619 02:44:18.656545 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.656597 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.656826 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.660569 (+ 3743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.660577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.660605 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.660961 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.660972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.662802 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.662808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.665000 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.665013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.666965 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.666970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.667162 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.667256 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.667403 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.667616 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"thread_start_us":5216,"threads_started":1}
I20260619 02:44:18.669602 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.655351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33859 (local address 127.0.0.1:48308)
0619 02:44:18.660518 (+ 5167us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.660538 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.660663 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.661343 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.661347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.661372 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.661943 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.661960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.664594 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.664599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.666795 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.666809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.668874 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.668880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.669028 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.669062 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.669142 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.669260 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5019,"thread_start_us":3873,"threads_started":1}
I20260619 02:44:18.670470 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.658003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.7.246.67:33859)
0619 02:44:18.659675 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.659683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.660798 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 02:44:18.660897 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.660905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.661143 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.661601 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.662766 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.664449 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.666971 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.668651 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.669662 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.669887 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.669953 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.670099 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":1335,"threads_started":1}
I20260619 02:44:18.670502 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.661365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:38867 (local address 127.0.0.1:33996)
0619 02:44:18.662521 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.662539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.662710 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.663513 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.663517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.663546 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.663939 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.663952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.665356 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.665367 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.667708 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.667723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.669801 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.669808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.669969 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.669989 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.670092 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.670212 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":907,"threads_started":1}
I20260619 02:44:18.671568 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.660836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.7.246.66:38867)
0619 02:44:18.661311 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.661320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.662630 (+ 1310us) server_negotiation.cc:407] Connection header received
0619 02:44:18.662898 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.662906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.663138 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.663289 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.664161 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.665213 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.667940 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.669305 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.669436 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.669661 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.670158 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.670325 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260619 02:44:18.672360 9414 tablet_service.cc:1467] Processing CreateTablet for tablet bfac1494b06d451cb898d2ebd249596e (DEFAULT_TABLE table=test-table [id=ed942e8c43d44193a60208a42da3482d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.673843 9414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac1494b06d451cb898d2ebd249596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.674167 9553 tablet_service.cc:1467] Processing CreateTablet for tablet bfac1494b06d451cb898d2ebd249596e (DEFAULT_TABLE table=test-table [id=ed942e8c43d44193a60208a42da3482d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.674067 9692 tablet_service.cc:1467] Processing CreateTablet for tablet bfac1494b06d451cb898d2ebd249596e (DEFAULT_TABLE table=test-table [id=ed942e8c43d44193a60208a42da3482d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.675089 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac1494b06d451cb898d2ebd249596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.675381 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac1494b06d451cb898d2ebd249596e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.686121 9781 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Bootstrap starting.
I20260619 02:44:18.688967 9782 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Bootstrap starting.
I20260619 02:44:18.689697 9781 tablet_bootstrap.cc:654] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.690515 9783 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Bootstrap starting.
I20260619 02:44:18.692205 9781 log.cc:824] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:18.693001 9782 tablet_bootstrap.cc:654] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.693727 9783 tablet_bootstrap.cc:654] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.694118 9782 log.cc:824] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:18.694837 9783 log.cc:824] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:18.695309 9781 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: No bootstrap required, opened a new log
I20260619 02:44:18.695588 9781 ts_tablet_manager.cc:1397] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260619 02:44:18.696976 9782 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: No bootstrap required, opened a new log
I20260619 02:44:18.697237 9782 ts_tablet_manager.cc:1397] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:44:18.703833 9783 tablet_bootstrap.cc:492] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: No bootstrap required, opened a new log
I20260619 02:44:18.704102 9783 ts_tablet_manager.cc:1397] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 02:44:18.706784 9781 raft_consensus.cc:348] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.707224 9781 raft_consensus.cc:374] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.707317 9781 raft_consensus.cc:729] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd4d02899de4875a3f594fc5d50df5c, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.707945 9781 consensus_queue.cc:260] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.708226 9782 raft_consensus.cc:348] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.708640 9782 raft_consensus.cc:374] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.708742 9782 raft_consensus.cc:729] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29325d9a6d3e4a19a9f1d3fd0584cc72, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.709234 9623 heartbeater.cc:503] Master 127.7.246.126:43963 was elected leader, sending a full tablet report...
I20260619 02:44:18.709424 9782 consensus_queue.cc:260] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.709717 9781 ts_tablet_manager.cc:1428] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260619 02:44:18.712855 9782 ts_tablet_manager.cc:1428] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260619 02:44:18.713109 9783 raft_consensus.cc:348] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.713465 9783 raft_consensus.cc:374] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.713618 9783 raft_consensus.cc:729] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fd9e9d1590421b95d926a302f2cdaf, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.714226 9783 consensus_queue.cc:260] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.715880 9762 heartbeater.cc:503] Master 127.7.246.126:43963 was elected leader, sending a full tablet report...
I20260619 02:44:18.716564 9783 ts_tablet_manager.cc:1428] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
W20260619 02:44:18.806397 9485 tablet.cc:2369] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:18.807802 9788 raft_consensus.cc:482] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:18.808077 9788 raft_consensus.cc:504] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.809819 9788 leader_election.cc:288] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867), b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859)
W20260619 02:44:18.813385 9624 tablet.cc:2369] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:18.817189 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.810109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33859 (local address 127.7.246.65:38579)
0619 02:44:18.811090 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.811108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.811259 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.811552 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.811556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.811580 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.811969 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.811977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.812980 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.812985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.814958 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.814969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.816441 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.816448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.816618 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.816640 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.816711 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.816860 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":82,"thread_start_us":691,"threads_started":1}
I20260619 02:44:18.817233 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.810166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:57311 (local address 127.7.246.66:38867)
0619 02:44:18.810419 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.810428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.811149 (+ 721us) server_negotiation.cc:407] Connection header received
0619 02:44:18.811373 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.811376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.811430 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.811525 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.812097 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.812792 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.815139 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.816318 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.816476 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.816542 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.816855 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.816985 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:44:18.817265 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.810368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:38579 (local address 127.7.246.67:33859)
0619 02:44:18.810644 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.810652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.811196 (+ 544us) server_negotiation.cc:407] Connection header received
0619 02:44:18.811395 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.811399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.811450 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.811541 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.812109 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.812842 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.815105 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.815842 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.815960 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.816010 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.816892 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.817040 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 02:44:18.817919 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.810021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:38867 (local address 127.7.246.65:57311)
0619 02:44:18.811029 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.811045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.811225 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.811538 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.811542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.811565 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.811956 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.811965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.812917 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.812924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.814968 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.814979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.816539 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.816544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.816659 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.816681 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:44:18.817420 (+ 739us) client_negotiation.cc:240] Negotiation successful
0619 02:44:18.817578 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":143,"thread_start_us":607,"threads_started":1}
I20260619 02:44:18.818476 9717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac1494b06d451cb898d2ebd249596e" candidate_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" is_pre_election: true
I20260619 02:44:18.818888 9573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac1494b06d451cb898d2ebd249596e" candidate_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd4d02899de4875a3f594fc5d50df5c" is_pre_election: true
I20260619 02:44:18.819057 9717 raft_consensus.cc:2454] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29325d9a6d3e4a19a9f1d3fd0584cc72 in term 0.
I20260619 02:44:18.819402 9573 raft_consensus.cc:2454] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29325d9a6d3e4a19a9f1d3fd0584cc72 in term 0.
I20260619 02:44:18.819715 9367 leader_election.cc:302] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29325d9a6d3e4a19a9f1d3fd0584cc72, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:18.820106 9788 raft_consensus.cc:2781] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:18.820221 9788 raft_consensus.cc:482] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:18.820362 9788 raft_consensus.cc:3037] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:18.822598 9788 raft_consensus.cc:504] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.823236 9788 leader_election.cc:288] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [CANDIDATE]: Term 1 election: Requested vote from peers 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867), b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859)
I20260619 02:44:18.823664 9573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac1494b06d451cb898d2ebd249596e" candidate_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd4d02899de4875a3f594fc5d50df5c"
I20260619 02:44:18.823693 9717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac1494b06d451cb898d2ebd249596e" candidate_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fd9e9d1590421b95d926a302f2cdaf"
I20260619 02:44:18.823863 9573 raft_consensus.cc:3037] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:18.823881 9717 raft_consensus.cc:3037] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:18.826002 9717 raft_consensus.cc:2454] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29325d9a6d3e4a19a9f1d3fd0584cc72 in term 1.
I20260619 02:44:18.826020 9573 raft_consensus.cc:2454] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29325d9a6d3e4a19a9f1d3fd0584cc72 in term 1.
I20260619 02:44:18.826442 9367 leader_election.cc:302] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29325d9a6d3e4a19a9f1d3fd0584cc72, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:18.826741 9788 raft_consensus.cc:2781] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:18.827184 9788 raft_consensus.cc:686] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 LEADER]: Becoming Leader. State: Replica: 29325d9a6d3e4a19a9f1d3fd0584cc72, State: Running, Role: LEADER
I20260619 02:44:18.827685 9788 consensus_queue.cc:237] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
W20260619 02:44:18.828788 9763 tablet.cc:2369] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:18.832454 9277 catalog_manager.cc:5495] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65). New cstate: current_term: 1 leader_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:18.868003 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.862542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.7.246.65:33973)
0619 02:44:18.862730 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.862738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.862777 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:18.863134 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.863137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.863209 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.863341 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:18.863681 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.864388 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.865743 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.866149 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.866321 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:18.866796 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:18.867298 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:18.867448 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:18.867653 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 02:44:18.867781 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:44:18.876071 9717 raft_consensus.cc:1261] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Refusing update from remote peer 29325d9a6d3e4a19a9f1d3fd0584cc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:18.876189 9573 raft_consensus.cc:1261] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Refusing update from remote peer 29325d9a6d3e4a19a9f1d3fd0584cc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:18.877453 9792 consensus_queue.cc:1035] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 02:44:18.877832 9788 consensus_queue.cc:1035] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:18.893613 9795 mvcc.cc:205] Tried to move back new op lower bound from 7298404593147179008 to 7298404592961703936. Current Snapshot: MvccSnapshot[applied={T|T < 7298404593147179008}]
I20260619 02:44:18.927767 9276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:18.928081 9276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:18.932665 9276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Sending DeleteTablet for 3 replicas of tablet bfac1494b06d451cb898d2ebd249596e
I20260619 02:44:18.934068 9553 tablet_service.cc:1514] Processing DeleteTablet for tablet bfac1494b06d451cb898d2ebd249596e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:18 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:33996
I20260619 02:44:18.934183 9692 tablet_service.cc:1514] Processing DeleteTablet for tablet bfac1494b06d451cb898d2ebd249596e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:18 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:48308
I20260619 02:44:18.934942 9805 tablet_replica.cc:330] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: stopping tablet replica
I20260619 02:44:18.934875 9414 tablet_service.cc:1514] Processing DeleteTablet for tablet bfac1494b06d451cb898d2ebd249596e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:18 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53590
I20260619 02:44:18.935389 9805 raft_consensus.cc:2229] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:18.935985 9805 raft_consensus.cc:2258] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:18.937047 9276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:18.937723 9276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:18.938349 9805 ts_tablet_manager.cc:1905] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:18.941690 9806 tablet_replica.cc:330] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: stopping tablet replica
I20260619 02:44:18.942215 9806 raft_consensus.cc:2229] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:18.942819 9806 raft_consensus.cc:2258] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:18.945791 9805 ts_tablet_manager.cc:1918] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:18.945947 9805 log.cc:1196] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/bfac1494b06d451cb898d2ebd249596e
I20260619 02:44:18.946480 9805 ts_tablet_manager.cc:1939] T bfac1494b06d451cb898d2ebd249596e P 7bd4d02899de4875a3f594fc5d50df5c: Deleting consensus metadata
I20260619 02:44:18.946822 9691 tablet_service.cc:1467] Processing CreateTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 (DEFAULT_TABLE table=test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.947264 9691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b9f7a2557747d2929d1c812ac934b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.947466 9806 ts_tablet_manager.cc:1905] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:18.950080 9262 catalog_manager.cc:4841] TS 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867): tablet bfac1494b06d451cb898d2ebd249596e (table test-table [id=ed942e8c43d44193a60208a42da3482d]) successfully deleted
I20260619 02:44:18.950413 9807 tablet_replica.cc:330] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: stopping tablet replica
I20260619 02:44:18.951058 9413 tablet_service.cc:1467] Processing CreateTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 (DEFAULT_TABLE table=test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.951179 9553 tablet_service.cc:1467] Processing CreateTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 (DEFAULT_TABLE table=test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:18.951678 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b9f7a2557747d2929d1c812ac934b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.951943 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b9f7a2557747d2929d1c812ac934b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:18.956974 9806 ts_tablet_manager.cc:1918] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:18.957165 9806 log.cc:1196] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/bfac1494b06d451cb898d2ebd249596e
I20260619 02:44:18.957824 9806 ts_tablet_manager.cc:1939] T bfac1494b06d451cb898d2ebd249596e P b3fd9e9d1590421b95d926a302f2cdaf: Deleting consensus metadata
I20260619 02:44:18.958528 9782 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Bootstrap starting.
I20260619 02:44:18.959165 9807 raft_consensus.cc:2229] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:18.959682 9264 catalog_manager.cc:4841] TS b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859): tablet bfac1494b06d451cb898d2ebd249596e (table test-table [id=ed942e8c43d44193a60208a42da3482d]) successfully deleted
I20260619 02:44:18.959874 9807 raft_consensus.cc:2258] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:18.960588 9782 tablet_bootstrap.cc:654] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.961524 9807 ts_tablet_manager.cc:1905] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:18.968456 9807 ts_tablet_manager.cc:1918] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:18.968606 9807 log.cc:1196] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/bfac1494b06d451cb898d2ebd249596e
I20260619 02:44:18.969089 9807 ts_tablet_manager.cc:1939] T bfac1494b06d451cb898d2ebd249596e P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting consensus metadata
I20260619 02:44:18.969437 9783 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Bootstrap starting.
I20260619 02:44:18.970760 9783 tablet_bootstrap.cc:654] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.970745 9262 catalog_manager.cc:4841] TS 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973): tablet bfac1494b06d451cb898d2ebd249596e (table test-table [id=ed942e8c43d44193a60208a42da3482d]) successfully deleted
I20260619 02:44:18.971637 9781 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Bootstrap starting.
I20260619 02:44:18.972883 9783 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: No bootstrap required, opened a new log
I20260619 02:44:18.972899 9782 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: No bootstrap required, opened a new log
I20260619 02:44:18.973024 9783 ts_tablet_manager.cc:1397] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:44:18.973145 9782 ts_tablet_manager.cc:1397] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20260619 02:44:18.973173 9781 tablet_bootstrap.cc:654] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:18.974404 9783 raft_consensus.cc:348] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.974615 9783 raft_consensus.cc:374] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.974707 9783 raft_consensus.cc:729] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fd9e9d1590421b95d926a302f2cdaf, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.974999 9783 consensus_queue.cc:260] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.975052 9782 raft_consensus.cc:348] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.975386 9782 raft_consensus.cc:374] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.975524 9782 raft_consensus.cc:729] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29325d9a6d3e4a19a9f1d3fd0584cc72, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.975816 9782 consensus_queue.cc:260] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.976536 9782 ts_tablet_manager.cc:1428] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:18.976553 9783 ts_tablet_manager.cc:1428] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 02:44:18.980445 9781 tablet_bootstrap.cc:492] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: No bootstrap required, opened a new log
I20260619 02:44:18.980599 9781 ts_tablet_manager.cc:1397] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 02:44:18.981446 9781 raft_consensus.cc:348] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.981743 9781 raft_consensus.cc:374] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:18.981891 9781 raft_consensus.cc:729] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd4d02899de4875a3f594fc5d50df5c, State: Initialized, Role: FOLLOWER
I20260619 02:44:18.982148 9781 consensus_queue.cc:260] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.983348 9781 ts_tablet_manager.cc:1428] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:18.991600 9787 raft_consensus.cc:482] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:18.991815 9787 raft_consensus.cc:504] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:18.993534 9787 leader_election.cc:288] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973), b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859)
I20260619 02:44:19.002000 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.994794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:48161 (local address 127.7.246.67:33859)
0619 02:44:18.994967 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.994973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.995565 (+ 592us) server_negotiation.cc:407] Connection header received
0619 02:44:18.995764 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.995772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.995850 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.995953 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.996536 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.997493 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.999690 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.000848 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.000994 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.001063 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.001476 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.001663 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:44:19.002267 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.994004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:37183 (local address 127.7.246.65:33973)
0619 02:44:18.994505 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:18.994512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:18.994610 (+ 98us) server_negotiation.cc:407] Connection header received
0619 02:44:18.994844 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:18.994851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:18.994945 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:18.995043 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:18.995834 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.996995 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.999998 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.001251 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.001434 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.001500 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.001838 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.001954 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260619 02:44:19.002876 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.993991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33859 (local address 127.7.246.66:48161)
0619 02:44:18.995435 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.995453 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.995596 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.995963 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.995967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.995988 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.996359 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.996372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.997706 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.997709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.999548 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.999559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.001066 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.001072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.001265 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.001300 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.002274 (+ 974us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.002422 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"mutex_wait_us":24,"thread_start_us":1123,"threads_started":1}
I20260619 02:44:19.003175 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:18.993734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33973 (local address 127.7.246.66:37183)
0619 02:44:18.994498 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:18.994513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:18.994692 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:18.995243 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:18.995247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:18.995276 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:18.995636 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.995648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:18.997178 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:18.997184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:18.999830 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:18.999844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.001457 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.001463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.001677 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.001713 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.001811 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.001931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":1,"thread_start_us":499,"threads_started":1}
I20260619 02:44:19.003505 9716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b9f7a2557747d2929d1c812ac934b2" candidate_uuid: "7bd4d02899de4875a3f594fc5d50df5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" is_pre_election: true
I20260619 02:44:19.003832 9716 raft_consensus.cc:2454] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd4d02899de4875a3f594fc5d50df5c in term 0.
I20260619 02:44:19.004307 9507 leader_election.cc:302] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd4d02899de4875a3f594fc5d50df5c, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:19.004199 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b9f7a2557747d2929d1c812ac934b2" candidate_uuid: "7bd4d02899de4875a3f594fc5d50df5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" is_pre_election: true
I20260619 02:44:19.004753 9787 raft_consensus.cc:2781] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:19.004880 9787 raft_consensus.cc:482] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:19.004963 9787 raft_consensus.cc:3037] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.004654 9439 raft_consensus.cc:2454] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd4d02899de4875a3f594fc5d50df5c in term 0.
I20260619 02:44:19.007987 9787 raft_consensus.cc:504] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.008705 9787 leader_election.cc:288] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [CANDIDATE]: Term 1 election: Requested vote from peers 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973), b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859)
I20260619 02:44:19.009083 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b9f7a2557747d2929d1c812ac934b2" candidate_uuid: "7bd4d02899de4875a3f594fc5d50df5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72"
I20260619 02:44:19.009210 9716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b9f7a2557747d2929d1c812ac934b2" candidate_uuid: "7bd4d02899de4875a3f594fc5d50df5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fd9e9d1590421b95d926a302f2cdaf"
I20260619 02:44:19.009295 9439 raft_consensus.cc:3037] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.009429 9716 raft_consensus.cc:3037] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.012400 9716 raft_consensus.cc:2454] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bd4d02899de4875a3f594fc5d50df5c in term 1.
I20260619 02:44:19.012543 9439 raft_consensus.cc:2454] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bd4d02899de4875a3f594fc5d50df5c in term 1.
I20260619 02:44:19.012810 9507 leader_election.cc:302] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd4d02899de4875a3f594fc5d50df5c, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:19.013155 9787 raft_consensus.cc:2781] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:19.013700 9787 raft_consensus.cc:686] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 LEADER]: Becoming Leader. State: Replica: 7bd4d02899de4875a3f594fc5d50df5c, State: Running, Role: LEADER
I20260619 02:44:19.014176 9787 consensus_queue.cc:237] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.018242 9277 catalog_manager.cc:5495] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66). New cstate: current_term: 1 leader_uuid: "7bd4d02899de4875a3f594fc5d50df5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837056, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:19.068322 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.052438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43963 (local address 127.0.0.1:45454)
0619 02:44:19.053076 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.053096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.053256 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.053594 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.053600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.054503 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:19.054914 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.054940 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.055975 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.055981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.057444 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.057458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.058163 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.058169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.058337 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.059045 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:19.059069 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:19.061961 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:19.065638 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:19.065649 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:19.065657 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:19.066156 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:19.066513 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:19.066518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:19.066521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:19.066682 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:19.067325 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:19.067335 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:19.067497 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.067654 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.067897 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":304,"threads_started":1}
I20260619 02:44:19.069001 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.052420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.7.246.126:43963)
0619 02:44:19.052617 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.052625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.053195 (+ 570us) server_negotiation.cc:407] Connection header received
0619 02:44:19.053364 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.053368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.053441 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.053521 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.055097 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.055861 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.057605 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.057988 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.058081 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.062157 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.062163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.062168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.062204 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.065478 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.066278 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.066283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.066287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.066414 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.066775 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.066780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.066784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.067160 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.067296 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.068527 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.068699 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:44:19.074635 9277 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:45454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:44:19.075801 9277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:19.083114 9691 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea13d4775847ec928e5dd9a59c3bb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38831e3ab86b4991b3e62b5b68f10f67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:19.083328 9553 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea13d4775847ec928e5dd9a59c3bb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38831e3ab86b4991b3e62b5b68f10f67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:19.083639 9691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea13d4775847ec928e5dd9a59c3bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:19.083798 9553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea13d4775847ec928e5dd9a59c3bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:19.090059 9414 tablet_service.cc:1467] Processing CreateTablet for tablet 0dea13d4775847ec928e5dd9a59c3bb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38831e3ab86b4991b3e62b5b68f10f67]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:19.090566 9781 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: Bootstrap starting.
I20260619 02:44:19.090617 9414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dea13d4775847ec928e5dd9a59c3bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:19.091794 9781 tablet_bootstrap.cc:654] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:19.093464 9781 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: No bootstrap required, opened a new log
I20260619 02:44:19.093668 9781 ts_tablet_manager.cc:1397] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:19.094198 9781 raft_consensus.cc:348] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.094413 9781 raft_consensus.cc:374] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:19.094494 9781 raft_consensus.cc:729] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd4d02899de4875a3f594fc5d50df5c, State: Initialized, Role: FOLLOWER
I20260619 02:44:19.094748 9781 consensus_queue.cc:260] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.095105 9787 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.095266 9781 ts_tablet_manager.cc:1428] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:44:19.095404 9787 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.098013 9782 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Bootstrap starting.
I20260619 02:44:19.099769 9782 tablet_bootstrap.cc:654] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:19.100179 9783 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: Bootstrap starting.
I20260619 02:44:19.101863 9783 tablet_bootstrap.cc:654] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:19.103061 9782 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: No bootstrap required, opened a new log
I20260619 02:44:19.103273 9782 ts_tablet_manager.cc:1397] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 02:44:19.103907 9783 tablet_bootstrap.cc:492] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: No bootstrap required, opened a new log
I20260619 02:44:19.104146 9783 ts_tablet_manager.cc:1397] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260619 02:44:19.104032 9782 raft_consensus.cc:348] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.104261 9782 raft_consensus.cc:374] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:19.104351 9782 raft_consensus.cc:729] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29325d9a6d3e4a19a9f1d3fd0584cc72, State: Initialized, Role: FOLLOWER
I20260619 02:44:19.104631 9782 consensus_queue.cc:260] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.104748 9783 raft_consensus.cc:348] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.104976 9783 raft_consensus.cc:374] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:19.105069 9783 raft_consensus.cc:729] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fd9e9d1590421b95d926a302f2cdaf, State: Initialized, Role: FOLLOWER
I20260619 02:44:19.105016 9792 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.105329 9792 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.105288 9783 consensus_queue.cc:260] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.105913 9783 ts_tablet_manager.cc:1428] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:44:19.105764 9789 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.107430 9789 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.109028 9782 ts_tablet_manager.cc:1428] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20260619 02:44:19.287446 9789 raft_consensus.cc:482] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:19.287716 9789 raft_consensus.cc:504] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.289594 9789 leader_election.cc:288] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973), 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867)
I20260619 02:44:19.297950 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.291127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:34925 (local address 127.7.246.66:38867)
0619 02:44:19.291361 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.291370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.291967 (+ 597us) server_negotiation.cc:407] Connection header received
0619 02:44:19.292250 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.292256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.292336 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.292431 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:19.293018 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.294184 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.296128 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.296895 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.297077 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.297159 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.297497 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.297662 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:44:19.297971 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.291005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:38867 (local address 127.7.246.67:34925)
0619 02:44:19.291852 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.291866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.292016 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.292462 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.292467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.292494 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:19.292830 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.292843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.294326 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.294333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.295963 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.295974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.297077 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.297084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.297319 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.297366 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.297468 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.297626 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":570,"threads_started":1}
I20260619 02:44:19.298247 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.290059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:47017 (local address 127.7.246.65:33973)
0619 02:44:19.290283 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.290290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.290974 (+ 684us) server_negotiation.cc:407] Connection header received
0619 02:44:19.291276 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.291283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.291369 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.291439 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:19.292153 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.293219 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.295396 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.297130 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.297307 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.297383 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.297782 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.297923 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:44:19.298787 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.290290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33973 (local address 127.7.246.67:47017)
0619 02:44:19.290867 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.290890 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.291085 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.291655 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.291660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.291691 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:19.291960 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.291974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.293355 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.293362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.295213 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.295228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.297366 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.297372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.297586 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.297627 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.298285 (+ 658us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.298459 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1,"thread_start_us":280,"threads_started":1}
I20260619 02:44:19.298996 9573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea13d4775847ec928e5dd9a59c3bb9" candidate_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd4d02899de4875a3f594fc5d50df5c" is_pre_election: true
I20260619 02:44:19.299300 9573 raft_consensus.cc:2454] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3fd9e9d1590421b95d926a302f2cdaf in term 0.
I20260619 02:44:19.299513 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea13d4775847ec928e5dd9a59c3bb9" candidate_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" is_pre_election: true
I20260619 02:44:19.299825 9439 raft_consensus.cc:2454] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3fd9e9d1590421b95d926a302f2cdaf in term 0.
I20260619 02:44:19.299817 9644 leader_election.cc:302] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd4d02899de4875a3f594fc5d50df5c, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:19.300235 9789 raft_consensus.cc:2781] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:19.300367 9789 raft_consensus.cc:482] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:19.300480 9789 raft_consensus.cc:3037] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.302662 9789 raft_consensus.cc:504] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.303309 9789 leader_election.cc:288] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [CANDIDATE]: Term 1 election: Requested vote from peers 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973), 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867)
I20260619 02:44:19.303741 9439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea13d4775847ec928e5dd9a59c3bb9" candidate_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72"
I20260619 02:44:19.303918 9573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dea13d4775847ec928e5dd9a59c3bb9" candidate_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd4d02899de4875a3f594fc5d50df5c"
I20260619 02:44:19.304445 9439 raft_consensus.cc:3037] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.304525 9573 raft_consensus.cc:3037] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:19.306933 9439 raft_consensus.cc:2454] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3fd9e9d1590421b95d926a302f2cdaf in term 1.
I20260619 02:44:19.307363 9644 leader_election.cc:302] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29325d9a6d3e4a19a9f1d3fd0584cc72, b3fd9e9d1590421b95d926a302f2cdaf; no voters:
I20260619 02:44:19.307667 9789 raft_consensus.cc:2781] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:19.307699 9573 raft_consensus.cc:2454] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3fd9e9d1590421b95d926a302f2cdaf in term 1.
I20260619 02:44:19.308117 9789 raft_consensus.cc:686] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 LEADER]: Becoming Leader. State: Replica: b3fd9e9d1590421b95d926a302f2cdaf, State: Running, Role: LEADER
I20260619 02:44:19.308560 9789 consensus_queue.cc:237] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } }
I20260619 02:44:19.310653 9826 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: TxnStatusTablet state changed. Reason: New leader b3fd9e9d1590421b95d926a302f2cdaf. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.311012 9826 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:19.312238 9828 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:44:19.312469 9828 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:44:19.312742 9277 catalog_manager.cc:5495] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf reported cstate change: term changed from 0 to 1, leader changed from <none> to b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67). New cstate: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } health_report { overall_health: HEALTHY } } }
I20260619 02:44:19.354444 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.345613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.7.246.67:33859)
0619 02:44:19.345795 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.345801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.346584 (+ 783us) server_negotiation.cc:407] Connection header received
0619 02:44:19.346934 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.346943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.347053 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.347217 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:19.347687 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.348890 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.350901 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.352125 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.352251 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.352887 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.353703 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.353856 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.354012 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.354110 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:44:19.354522 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.345688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33859 (local address 127.0.0.1:48316)
0619 02:44:19.346463 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.346485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.346668 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.347269 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.347274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.347290 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:19.347545 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.347559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.349071 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.349078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.350746 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.350764 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.352288 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.352297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.352544 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.352643 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.353840 (+ 1197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.353866 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.353976 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.354159 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592}
I20260619 02:44:19.359618 9573 raft_consensus.cc:1261] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Refusing update from remote peer b3fd9e9d1590421b95d926a302f2cdaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:19.360435 9439 raft_consensus.cc:1261] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Refusing update from remote peer b3fd9e9d1590421b95d926a302f2cdaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:19.361337 9789 consensus_queue.cc:1035] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:19.361686 9826 consensus_queue.cc:1035] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:19.373224 9792 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: TxnStatusTablet state changed. Reason: New leader b3fd9e9d1590421b95d926a302f2cdaf. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.373539 9792 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.376494 9826 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.376852 9826 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:19.377420 9796 mvcc.cc:205] Tried to move back new op lower bound from 7298404595128205312 to 7298404594931953664. Current Snapshot: MvccSnapshot[applied={T|T < 7298404595128205312 or (T in {7298404595128205312})}]
I20260619 02:44:19.379863 9792 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.380117 9792 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 29325d9a6d3e4a19a9f1d3fd0584cc72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.380210 9787 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: TxnStatusTablet state changed. Reason: New leader b3fd9e9d1590421b95d926a302f2cdaf. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.380556 9787 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.388820 9789 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.389130 9789 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P b3fd9e9d1590421b95d926a302f2cdaf: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:19.395033 9787 tablet_replica.cc:439] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 } } peers { permanent_uuid: "7bd4d02899de4875a3f594fc5d50df5c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 38867 } } peers { permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 } } }
I20260619 02:44:19.395362 9787 tablet_replica.cc:442] T 0dea13d4775847ec928e5dd9a59c3bb9 P 7bd4d02899de4875a3f594fc5d50df5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:19.412485 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.397999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.7.246.66:38867)
0619 02:44:19.398155 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.398160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.405186 (+ 7026us) server_negotiation.cc:407] Connection header received
0619 02:44:19.405666 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.405672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.405793 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.405934 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:19.406641 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.407749 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.409763 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.410419 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.410512 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.411122 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.411800 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.411901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.412125 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.412205 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:44:19.427663 9787 consensus_queue.cc:1035] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "29325d9a6d3e4a19a9f1d3fd0584cc72" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:19.433970 9787 consensus_queue.cc:1035] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fd9e9d1590421b95d926a302f2cdaf" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:19.445674 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.425601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33859 (local address 127.7.246.66:56097)
0619 02:44:19.437979 (+ 12378us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.438001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.438468 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.438988 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.438993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.439005 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:19.439373 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.439382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.440904 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.440910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.442816 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.442831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.444377 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.444382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.444504 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.444573 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.445080 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.445094 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.445168 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.445277 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12188,"thread_start_us":12453,"threads_started":1}
I20260619 02:44:19.445677 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.425755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:56097 (local address 127.7.246.67:33859)
0619 02:44:19.426013 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.426019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.438398 (+ 12379us) server_negotiation.cc:407] Connection header received
0619 02:44:19.438605 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.438610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.438720 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.438837 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:19.439527 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.440685 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.442970 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.444188 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.444337 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.444792 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.444931 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.445063 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.445221 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.445310 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 02:44:19.485481 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.478711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:54961 (local address 127.7.246.66:38867)
0619 02:44:19.478900 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.478906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.479085 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:44:19.479388 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.479394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.479476 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.479595 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:19.480033 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.481286 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.482993 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.483654 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.483807 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.484282 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.484405 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.484509 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.484677 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.484750 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:44:19.485953 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.478470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:38867 (local address 127.7.246.67:54961)
0619 02:44:19.478987 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:19.479010 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:19.479218 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:19.479589 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:19.479592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:19.479601 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:19.479877 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.479887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.481427 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.481432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.482844 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:19.482856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.483859 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.483868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:19.484015 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.484072 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:19.484525 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:19.484536 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:44:19.484656 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:44:19.484792 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":438,"threads_started":1}
I20260619 02:44:19.527333 9277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:19.527647 9277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:19.530854 9277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f92b2897b2a7445bbbc86b548581321b: Sending DeleteTablet for 3 replicas of tablet b1b9f7a2557747d2929d1c812ac934b2
I20260619 02:44:19.531793 9553 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:33996
I20260619 02:44:19.532598 9414 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53590
I20260619 02:44:19.533628 9691 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b9f7a2557747d2929d1c812ac934b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:48308
I20260619 02:44:19.538194 9850 tablet_replica.cc:330] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: stopping tablet replica
I20260619 02:44:19.539451 9850 raft_consensus.cc:2229] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:19.539757 9850 raft_consensus.cc:2258] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:19.544196 9848 tablet_replica.cc:330] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: stopping tablet replica
I20260619 02:44:19.544462 9848 raft_consensus.cc:2229] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:19.544723 9848 raft_consensus.cc:2258] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:19.547674 9850 ts_tablet_manager.cc:1905] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:19.547732 9848 ts_tablet_manager.cc:1905] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:19.549772 9847 tablet_replica.cc:330] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: stopping tablet replica
I20260619 02:44:19.550136 9847 raft_consensus.cc:2229] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:19.550796 9847 raft_consensus.cc:2258] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:19.558686 9850 ts_tablet_manager.cc:1918] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:19.558841 9850 log.cc:1196] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/b1b9f7a2557747d2929d1c812ac934b2
I20260619 02:44:19.559319 9850 ts_tablet_manager.cc:1939] T b1b9f7a2557747d2929d1c812ac934b2 P b3fd9e9d1590421b95d926a302f2cdaf: Deleting consensus metadata
I20260619 02:44:19.560480 9264 catalog_manager.cc:4841] TS b3fd9e9d1590421b95d926a302f2cdaf (127.7.246.67:33859): tablet b1b9f7a2557747d2929d1c812ac934b2 (table test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]) successfully deleted
I20260619 02:44:19.560741 9848 ts_tablet_manager.cc:1918] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:19.560870 9848 log.cc:1196] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/b1b9f7a2557747d2929d1c812ac934b2
I20260619 02:44:19.561307 9848 ts_tablet_manager.cc:1939] T b1b9f7a2557747d2929d1c812ac934b2 P 29325d9a6d3e4a19a9f1d3fd0584cc72: Deleting consensus metadata
I20260619 02:44:19.566352 9847 ts_tablet_manager.cc:1905] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:19.568939 9262 catalog_manager.cc:4841] TS 29325d9a6d3e4a19a9f1d3fd0584cc72 (127.7.246.65:33973): tablet b1b9f7a2557747d2929d1c812ac934b2 (table test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]) successfully deleted
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:19.584193 9847 ts_tablet_manager.cc:1918] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:19.585641 9847 log.cc:1196] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/b1b9f7a2557747d2929d1c812ac934b2
I20260619 02:44:19.586520 9847 ts_tablet_manager.cc:1939] T b1b9f7a2557747d2929d1c812ac934b2 P 7bd4d02899de4875a3f594fc5d50df5c: Deleting consensus metadata
I20260619 02:44:19.590142 9262 catalog_manager.cc:4841] TS 7bd4d02899de4875a3f594fc5d50df5c (127.7.246.66:38867): tablet b1b9f7a2557747d2929d1c812ac934b2 (table test-table [id=c9eca5604c9c4bd7aacad1a0de910c3d]) successfully deleted
I20260619 02:44:19.590487 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.548297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.7.246.65:33973)
0619 02:44:19.548545 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.548554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.562501 (+ 13947us) server_negotiation.cc:407] Connection header received
0619 02:44:19.562968 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.562974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.563056 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.563117 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.566186 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.567198 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.570445 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.571089 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.571337 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.582018 (+ 10681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.582037 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.582056 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.582113 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.586320 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.587296 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.587310 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.587316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.587915 (+ 599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.588383 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.588389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.588393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.588878 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.589188 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.589850 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.590106 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 02:44:19.592082 9479 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53618
I20260619 02:44:19.646931 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.615080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.7.246.126:43963)
0619 02:44:19.629339 (+ 14259us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.629353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.630363 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 02:44:19.630445 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.630451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.630550 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.630668 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.632550 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.633780 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.636165 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.636922 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.637080 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.640140 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.640148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.640153 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.640199 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.643927 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.644758 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.644763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.644767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.644912 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.645321 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.645326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.645330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.645727 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.645994 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.646297 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.646523 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14092,"thread_start_us":14279,"threads_started":1}
W20260619 02:44:19.647609 9277 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45470
I20260619 02:44:19.649952 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9348
I20260619 02:44:19.671288 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.652268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.7.246.65:33973)
0619 02:44:19.652467 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.652473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.654255 (+ 1782us) server_negotiation.cc:407] Connection header received
0619 02:44:19.654456 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.654461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.654545 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.654647 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.656304 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.657457 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.659479 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.660150 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.660255 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.663701 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.663708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.663713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.663758 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.667408 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.668819 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.668825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.668830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.669005 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.669511 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.669517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.669520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.669894 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.670155 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.670502 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.670741 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 02:44:19.671841 9479 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53624
W20260619 02:44:19.672467 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:19.672667 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9348
I20260619 02:44:19.704744 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9488
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:19.725886 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.705656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.7.246.66:38867)
0619 02:44:19.705847 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.705853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.706005 (+ 152us) server_negotiation.cc:407] Connection header received
0619 02:44:19.706219 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.706225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.706311 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.706452 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.708171 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.709493 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.711442 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.712321 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.712468 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.718082 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.718103 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.718123 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.718178 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.722476 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.723466 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.723477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.723481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.723652 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.724136 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.724141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.724145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.724589 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.724913 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.725244 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.725461 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 02:44:19.726800 9618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34008
W20260619 02:44:19.727527 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9488: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:19.727782 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9488
I20260619 02:44:19.746681 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9627
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837058, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:19.764612 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:19.747499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.7.246.67:33859)
0619 02:44:19.747679 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:19.747685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:19.747819 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:44:19.748029 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:19.748035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:19.748135 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:19.748253 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:19.749450 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.750254 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.751500 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:19.751938 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:19.752100 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:19.756827 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:19.756846 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:19.756869 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:19.756934 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:19.761451 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.762241 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.762253 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.762256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.762425 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:19.762790 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:19.762796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:19.762800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:19.763320 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:19.763742 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:19.763910 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:44:19.764159 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 02:44:19.765355 9757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48322
W20260619 02:44:19.766211 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9627: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:19.766363 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9627
I20260619 02:44:19.786536 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9245
I20260619 02:44:19.787768 9317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:19.901026 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9245
2026-06-19T02:44:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4688 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9859](info): set up 2 sockets
Jun 19 02:44:19 dist-test-slave-9tkz krb5kdc[9859](info): commencing operation
krb5kdc: starting...
W20260619 02:44:20.989873 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:21 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:21.176759 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40137
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:21.497054 9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:21.497349 9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:21.497438 9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:21.504834 9872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:21.504958 9872 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:44:21.504999 9872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:21.505087 9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:21.505160 9872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:21.505268 9872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:21.515280 9872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:40137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:21.516675 9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:21.518029 9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:21.530049 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:21.530027 9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:21.530092 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:21.530727 9872 server_base.cc:1034] running on GCE node
I20260619 02:44:21.531584 9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:21.533046 9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:21.534271 9872 hybrid_clock.cc:648] HybridClock initialized: now 1781837061534193 us; error 59 us; skew 500 ppm
I20260619 02:44:21.534597 9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:21 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837061, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:21.537698 9872 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:21.539263 9872 webserver.cc:460] Webserver started at http://127.7.246.126:45565/ using document root <none> and password file <none>
I20260619 02:44:21.539987 9872 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:21.540117 9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:21.540431 9872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:21.543592 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "30f0169df1b147ee91594f6191c3a3a8"
format_stamp: "Formatted at 2026-06-19 02:44:21 on dist-test-slave-9tkz"
server_key: "57c8da4829f5bc9ca28535b4b184deb0"
server_key_iv: "5f6ccf79865396087daa450ac38514be"
server_key_version: "encryptionkey@0"
I20260619 02:44:21.544384 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "30f0169df1b147ee91594f6191c3a3a8"
format_stamp: "Formatted at 2026-06-19 02:44:21 on dist-test-slave-9tkz"
server_key: "57c8da4829f5bc9ca28535b4b184deb0"
server_key_iv: "5f6ccf79865396087daa450ac38514be"
server_key_version: "encryptionkey@0"
I20260619 02:44:21.549621 9872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:44:21.553010 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:21.554435 9872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:21.554616 9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "30f0169df1b147ee91594f6191c3a3a8"
format_stamp: "Formatted at 2026-06-19 02:44:21 on dist-test-slave-9tkz"
server_key: "57c8da4829f5bc9ca28535b4b184deb0"
server_key_iv: "5f6ccf79865396087daa450ac38514be"
server_key_version: "encryptionkey@0"
I20260619 02:44:21.554977 9872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:21.573292 9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:21.577293 9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:21.577620 9872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:21.598812 9872 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:40137
I20260619 02:44:21.598836 9948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:40137 every 8 connection(s)
I20260619 02:44:21.600171 9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:21.603830 9949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:21.604490 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9872
I20260619 02:44:21.604681 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:21.605337 8153 external_mini_cluster.cc:1428] Setting key 7de2f06203df96b688af1f9e9baef49a
I20260619 02:44:21.617246 9949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Bootstrap starting.
Jun 19 02:44:21 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:21.620792 9949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:21.621894 9949 log.cc:824] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:21.624681 9949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: No bootstrap required, opened a new log
I20260619 02:44:21.627977 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:21.607249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.7.246.126:40137)
0619 02:44:21.608024 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:21.608036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:21.608073 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:44:21.609701 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:21.609723 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:21.610104 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:21.610431 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:21.612134 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:21.613374 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:21.615153 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:21.615799 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:21.615925 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:21.620642 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:21.620650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:21.620664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:21.620710 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:21.624041 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:21.624708 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:21.624715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:21.624721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:21.624940 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:21.625310 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:21.625315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:21.625319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:21.625874 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:21.626208 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:21.626589 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 02:44:21.627003 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":362,"threads_started":1}
I20260619 02:44:21.633240 9949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } }
I20260619 02:44:21.633647 9949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:21.633756 9949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f0169df1b147ee91594f6191c3a3a8, State: Initialized, Role: FOLLOWER
I20260619 02:44:21.634281 9949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } }
I20260619 02:44:21.634517 9949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:21.634655 9949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:21.634828 9949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:21.636904 9949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } }
I20260619 02:44:21.637372 9949 leader_election.cc:302] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f0169df1b147ee91594f6191c3a3a8; no voters:
I20260619 02:44:21.637930 9949 leader_election.cc:288] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:21.638067 9954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:21.638794 9954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [term 1 LEADER]: Becoming Leader. State: Replica: 30f0169df1b147ee91594f6191c3a3a8, State: Running, Role: LEADER
I20260619 02:44:21.639335 9954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } }
I20260619 02:44:21.639998 9949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:21.642323 9955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f0169df1b147ee91594f6191c3a3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } } }
I20260619 02:44:21.642721 9955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:21.643344 9956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f0169df1b147ee91594f6191c3a3a8. Latest consensus state: current_term: 1 leader_uuid: "30f0169df1b147ee91594f6191c3a3a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f0169df1b147ee91594f6191c3a3a8" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40137 } } }
I20260619 02:44:21.643671 9956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:21.644347 9960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:21.653154 9960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:21.653944 9971 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:44:21.662231 9960 catalog_manager.cc:1347] Generated new cluster ID: e1787b209c924bfe990d828ad3cfc219
I20260619 02:44:21.662362 9960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:21.671548 9960 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:21.672811 9960 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:21.684051 9960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Generated new TSK 0
I20260619 02:44:21.684921 9960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:21.738550 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40137
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:22.058843 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:22.059128 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:22.059301 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:22.066624 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:22.066788 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:22.066946 9975 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:44:22.067030 9975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:22.067147 9975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:22.076684 9975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:22.077979 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:22.079291 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:22.092098 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:22.092156 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:22.092132 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:22.093331 9975 server_base.cc:1034] running on GCE node
I20260619 02:44:22.093997 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:22.094856 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:22.096082 9975 hybrid_clock.cc:648] HybridClock initialized: now 1781837062096018 us; error 82 us; skew 500 ppm
I20260619 02:44:22.096413 9975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:22 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:22.099436 9975 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:22.100847 9975 webserver.cc:460] Webserver started at http://127.7.246.65:42155/ using document root <none> and password file <none>
I20260619 02:44:22.101504 9975 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:22.101667 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:22.101972 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:22.104712 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "80c4ecf34f0c49d78820302bc2d95b17"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "f4fa30ebd0d332e21b626accfe180861"
server_key_iv: "e13102aabb78cf79d545cf9dfb8075c7"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.105463 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "80c4ecf34f0c49d78820302bc2d95b17"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "f4fa30ebd0d332e21b626accfe180861"
server_key_iv: "e13102aabb78cf79d545cf9dfb8075c7"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.110375 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:44:22.113704 9990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:22.114903 9975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:44:22.115098 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "80c4ecf34f0c49d78820302bc2d95b17"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "f4fa30ebd0d332e21b626accfe180861"
server_key_iv: "e13102aabb78cf79d545cf9dfb8075c7"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.115316 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:22.127511 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:22.131297 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:22.131520 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:22.134050 9975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:22.134162 9975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:22.134284 9975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:22.134382 9975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:22 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:22.157640 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.134528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:56383 (local address 127.7.246.126:40137)
0619 02:44:22.136894 (+ 2366us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:22.136901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:22.136926 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:44:22.139338 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:22.139344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:22.139440 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:22.139580 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:22.142124 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.143414 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.145261 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.146094 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.146244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.150232 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:22.150242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:22.150248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:22.150302 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:22.154097 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.154917 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.154926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.154932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.155104 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.155704 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.155711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.155716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.156122 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:22.156281 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:22.157011 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 02:44:22.157228 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2241,"thread_start_us":2146,"threads_started":1}
I20260619 02:44:22.159142 10000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.135196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.65:56383)
0619 02:44:22.136369 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:22.136432 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:22.138952 (+ 2520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:22.139970 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:22.139987 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:22.140901 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:22.141860 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.141883 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.143617 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.143623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.145104 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.145118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.146406 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.146413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.146608 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.147366 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:22.147389 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:22.149949 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:22.154267 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.154274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.154290 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.154761 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.155266 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.155274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.155279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.155555 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.156258 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:22.156270 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:22.156636 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 02:44:22.156909 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 02:44:22.157467 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":391,"threads_started":1}
I20260619 02:44:22.183048 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.169100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.65:52043)
0619 02:44:22.169887 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:22.169915 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:22.170087 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:22.170456 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:22.170460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:22.170948 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:22.171231 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.171240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.172266 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.172273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.173297 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.173308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.174254 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.174265 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.174561 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.175393 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:22.175425 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:22.176366 (+ 941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:22.180239 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.180248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.180252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.180896 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.181286 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.181291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.181295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.181430 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.181981 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:22.181986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:22.182236 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 02:44:22.182436 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 02:44:22.182731 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
I20260619 02:44:22.183501 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.168994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52043 (local address 127.7.246.126:40137)
0619 02:44:22.169182 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:22.169186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:22.170009 (+ 823us) server_negotiation.cc:407] Connection header received
0619 02:44:22.170247 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:22.170255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:22.170327 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:22.170431 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:22.171376 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.172119 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.173440 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.174010 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.174168 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.176562 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:22.176566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:22.176570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:22.176600 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:22.180032 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.181027 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.181034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.181038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.181169 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.181584 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.181590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.181593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.181857 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:22.182017 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:22.182814 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 02:44:22.183100 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:44:22.192134 9975 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33407
I20260619 02:44:22.192174 10110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33407 every 8 connection(s)
I20260619 02:44:22.193343 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:22.195466 10111 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40137
I20260619 02:44:22.195696 10111 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:22.196357 10111 heartbeater.cc:511] Master 127.7.246.126:40137 requested a full tablet report, sending...
I20260619 02:44:22.197116 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 9975
I20260619 02:44:22.197346 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:22.198179 8153 external_mini_cluster.cc:1428] Setting key ded01ac1faf918c8314840e6d432224b
I20260619 02:44:22.198735 9904 ts_manager.cc:194] Registered new tserver with Master: 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407)
I20260619 02:44:22.200446 9904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:56383
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:22.238022 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40137
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:22.560057 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:22.560362 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:22.560537 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:22.567698 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:22.567853 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:22.568008 10115 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:44:22.568092 10115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:22.568207 10115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:22.578015 10115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:22.579327 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:22.580827 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:22.596181 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:22.596189 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:22.596392 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:22.596971 10115 server_base.cc:1034] running on GCE node
I20260619 02:44:22.597704 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:22.598687 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:22.599907 10115 hybrid_clock.cc:648] HybridClock initialized: now 1781837062599843 us; error 63 us; skew 500 ppm
I20260619 02:44:22.600219 10115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:22 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:22.603392 10115 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:22.610788 10115 webserver.cc:460] Webserver started at http://127.7.246.66:41897/ using document root <none> and password file <none>
I20260619 02:44:22.611526 10115 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:22.611660 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:22.611989 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:22.615244 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "a0f37f6871b645a28dc31c4f266817c9"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "5741585c560081368a3bfd09dce5eea5"
server_key_iv: "e95d56943c0182288624cb468f0fb62a"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.616020 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "a0f37f6871b645a28dc31c4f266817c9"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "5741585c560081368a3bfd09dce5eea5"
server_key_iv: "e95d56943c0182288624cb468f0fb62a"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.621527 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:44:22.625234 10130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:22.626657 10115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:44:22.626837 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "a0f37f6871b645a28dc31c4f266817c9"
format_stamp: "Formatted at 2026-06-19 02:44:22 on dist-test-slave-9tkz"
server_key: "5741585c560081368a3bfd09dce5eea5"
server_key_iv: "e95d56943c0182288624cb468f0fb62a"
server_key_version: "encryptionkey@0"
I20260619 02:44:22.627027 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:22.641633 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:22.645900 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:22.646180 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:22.648772 10115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:22.648916 10115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:22.649048 10115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:22.649143 10115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:22 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837062, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:22.674054 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.649710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:60475 (local address 127.7.246.126:40137)
0619 02:44:22.649905 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:22.649910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:22.651425 (+ 1515us) server_negotiation.cc:407] Connection header received
0619 02:44:22.653512 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:22.653518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:22.653630 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:22.653774 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:22.655397 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.656718 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.659267 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.659955 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.660098 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.665749 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:22.665769 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:22.665777 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:22.665835 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:22.669278 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.670379 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.670384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.670386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.670563 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.671324 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.671329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.671333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.671787 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:22.671957 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:22.673156 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0619 02:44:22.673478 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:44:22.675246 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.650410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.66:60475)
0619 02:44:22.651259 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:22.651300 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:22.653158 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:22.653992 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:22.654000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:22.654620 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:22.655227 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.655244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.656908 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.656914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.659094 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.659113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.660263 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.660273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.660637 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.661512 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:22.661568 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:22.665416 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:22.669451 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.669464 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.669491 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.670193 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.670735 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.670742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.670751 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.671115 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.672040 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:22.672049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:22.672627 (+ 578us) client_negotiation.cc:769] Sending connection context
0619 02:44:22.672971 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 02:44:22.673591 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":370,"threads_started":1}
I20260619 02:44:22.702672 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.681358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:58993 (local address 127.7.246.126:40137)
0619 02:44:22.681615 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:22.681626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:22.682577 (+ 951us) server_negotiation.cc:407] Connection header received
0619 02:44:22.682771 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:22.682775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:22.682842 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:22.682971 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:22.686292 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.687636 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.690530 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.691323 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.691465 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.694974 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:22.694979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:22.694982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:22.695012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:22.698636 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.700000 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.700007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.700010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.700187 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:22.700727 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:22.700735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:22.700740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:22.701227 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:22.701369 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:22.701898 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 02:44:22.702122 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:44:22.703363 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:22.681338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.66:58993)
0619 02:44:22.682440 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:22.682466 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:22.682643 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:22.683008 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:22.683014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:22.685375 (+ 2361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:22.686033 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.686053 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.687917 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.687926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.690274 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:22.690302 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:22.691887 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:22.691899 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:22.692186 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:22.693252 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:22.693297 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:22.694743 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:22.698810 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.698820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.698827 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.699736 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.700329 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:22.700337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:22.700349 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:22.700572 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:22.701382 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:22.701388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:22.701627 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 02:44:22.701810 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 02:44:22.702275 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":742,"threads_started":1}
I20260619 02:44:22.703665 10115 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:42825
I20260619 02:44:22.703681 10249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:42825 every 8 connection(s)
I20260619 02:44:22.705256 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:22.707460 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10115
I20260619 02:44:22.707664 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:22.707923 10250 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40137
I20260619 02:44:22.708230 10250 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:22.708278 8153 external_mini_cluster.cc:1428] Setting key 7d6b72767c2aab1ca011d723f6cfc48f
I20260619 02:44:22.709115 10250 heartbeater.cc:511] Master 127.7.246.126:40137 requested a full tablet report, sending...
I20260619 02:44:22.710659 9904 ts_manager.cc:194] Registered new tserver with Master: a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825)
I20260619 02:44:22.711453 9904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:60475
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:22.749166 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40137
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:23.082314 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:23.082597 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:23.082755 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:23.089944 10254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:23.090049 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:23.090211 10254 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:44:23.090293 10254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:44:23.090432 10254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:23.100142 10254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:23.101447 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:23.102842 10254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:23.115787 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:23.115815 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:23.115844 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:23.117024 10254 server_base.cc:1034] running on GCE node
I20260619 02:44:23.117815 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:23.118816 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:23.120024 10254 hybrid_clock.cc:648] HybridClock initialized: now 1781837063119960 us; error 61 us; skew 500 ppm
I20260619 02:44:23.120373 10254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:23 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:23.128317 10254 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:23.129961 10254 webserver.cc:460] Webserver started at http://127.7.246.67:44875/ using document root <none> and password file <none>
I20260619 02:44:23.130750 10254 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:23.130894 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:23.131212 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:23.134195 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "01615a0d99bd4febb644911cedf39f3d"
format_stamp: "Formatted at 2026-06-19 02:44:23 on dist-test-slave-9tkz"
server_key: "8ebed5f35db1484f36fce100f5140899"
server_key_iv: "3a66cdc0910007a23c1c8b6e597a38f1"
server_key_version: "encryptionkey@0"
I20260619 02:44:23.134930 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "01615a0d99bd4febb644911cedf39f3d"
format_stamp: "Formatted at 2026-06-19 02:44:23 on dist-test-slave-9tkz"
server_key: "8ebed5f35db1484f36fce100f5140899"
server_key_iv: "3a66cdc0910007a23c1c8b6e597a38f1"
server_key_version: "encryptionkey@0"
I20260619 02:44:23.139897 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:23.143433 10269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:23.144876 10254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:23.145117 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "01615a0d99bd4febb644911cedf39f3d"
format_stamp: "Formatted at 2026-06-19 02:44:23 on dist-test-slave-9tkz"
server_key: "8ebed5f35db1484f36fce100f5140899"
server_key_iv: "3a66cdc0910007a23c1c8b6e597a38f1"
server_key_version: "encryptionkey@0"
I20260619 02:44:23.145340 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:23.165536 10254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:23.169449 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:23.169791 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:23.172544 10254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:23.172739 10254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:23.172865 10254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:23.172945 10254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:44:23 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837063, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:23.200500 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.172919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:47005 (local address 127.7.246.126:40137)
0619 02:44:23.173091 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.173101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.174581 (+ 1480us) server_negotiation.cc:407] Connection header received
0619 02:44:23.176801 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.176806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.176898 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.177007 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:23.179360 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.180417 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.183451 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.184032 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.184168 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.188659 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:23.188669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:23.188677 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:23.188733 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:23.192849 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.194091 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.194100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.194106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.194318 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.194958 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.194966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.194971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.195549 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:23.195734 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.199688 (+ 3954us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.199996 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:44:23.200798 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.173666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.67:47005)
0619 02:44:23.174422 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.174457 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.176476 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.177240 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.177248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.178111 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:23.179177 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.179206 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.180569 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.180574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.183224 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.183245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.184293 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.184303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.184506 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.185294 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:23.185321 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:23.188352 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:23.193055 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.193064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.193086 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.193903 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.194474 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.194482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.194489 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.194799 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.195817 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:23.195828 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:23.196371 (+ 543us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.196782 (+ 411us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.197450 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":330,"threads_started":1}
I20260619 02:44:23.203182 10111 heartbeater.cc:503] Master 127.7.246.126:40137 was elected leader, sending a full tablet report...
I20260619 02:44:23.220950 10254 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:34235
I20260619 02:44:23.220985 10388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:34235 every 8 connection(s)
I20260619 02:44:23.222218 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:23.223264 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.208619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:35499 (local address 127.7.246.126:40137)
0619 02:44:23.209616 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.209625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.209647 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:44:23.209709 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.209715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.209795 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.209907 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:23.210723 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.211984 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.213614 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.214386 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.214541 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.217064 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:23.217072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:23.217078 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:23.217146 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:23.220432 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.221220 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.221224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.221226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.221335 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.221757 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.221764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.221767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.222037 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:23.222208 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.222654 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.222875 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813}
I20260619 02:44:23.223377 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.208732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.7.246.67:35499)
0619 02:44:23.209298 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.209314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.209473 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.209938 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.209941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.210309 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:23.210580 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.210588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.212120 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.212123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.213438 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.213463 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.214555 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.214560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.214680 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.215371 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:23.215404 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:23.216853 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:23.220561 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.220566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.220570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.221053 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.221450 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.221454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.221458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.221653 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.222166 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:23.222171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:23.222388 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.222559 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.222904 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":315,"threads_started":1}
I20260619 02:44:23.225998 10389 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40137
I20260619 02:44:23.226255 10389 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:23.227288 10389 heartbeater.cc:511] Master 127.7.246.126:40137 requested a full tablet report, sending...
I20260619 02:44:23.228509 9904 ts_manager.cc:194] Registered new tserver with Master: 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:23.229293 9904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:47005
I20260619 02:44:23.230069 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10254
I20260619 02:44:23.230237 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:23.230790 8153 external_mini_cluster.cc:1428] Setting key a494ffd9779b62651cd6cb2adf3e22b3
I20260619 02:44:23.233390 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:44:23 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:44:23 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:23.279354 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.265834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.7.246.126:40137)
0619 02:44:23.266057 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.266065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.266365 (+ 300us) server_negotiation.cc:407] Connection header received
0619 02:44:23.266575 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.266580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.266641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.266744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:23.268096 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.268852 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.270157 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.270572 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.270698 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.274411 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:23.274416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:23.274419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:23.274452 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:23.276832 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.277419 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.277424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.277426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.277594 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.277979 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.277984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.277988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.278243 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:23.278484 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.278796 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.279007 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:44:23.283145 9904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:23.289955 9904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:23.312827 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.303868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.7.246.65:33407)
0619 02:44:23.305177 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.305185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.305212 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:44:23.305325 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.305334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.305571 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.305751 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.306513 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.308009 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.310275 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.311547 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.311702 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.311916 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.312360 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.312534 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":1031,"threads_started":1}
I20260619 02:44:23.312942 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.303552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33407 (local address 127.0.0.1:39494)
0619 02:44:23.304140 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.304161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.304286 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.305915 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.305918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.305965 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.306320 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.306331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.308160 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.308165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.310091 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.310112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.311746 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.311752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.311942 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.312019 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.312170 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.312365 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":165,"thread_start_us":274,"threads_started":1}
I20260619 02:44:23.313086 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.301583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:34235 (local address 127.0.0.1:35814)
0619 02:44:23.302765 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.302845 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.303018 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.305323 (+ 2305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.305327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.305355 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.305846 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.305858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.307636 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.307641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.310067 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.310087 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.312243 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.312249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.312415 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.312446 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.312527 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.312648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":10,"thread_start_us":799,"threads_started":1}
I20260619 02:44:23.313302 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.302293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.7.246.66:42825)
0619 02:44:23.304488 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.304497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.304525 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:44:23.304620 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.304628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.304893 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.305663 (+ 770us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.305865 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.307155 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.310577 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.311934 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.312069 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.312245 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.312850 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.313021 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2034,"thread_start_us":2342,"threads_started":1}
I20260619 02:44:23.313375 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.301813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.0.0.1:43958)
0619 02:44:23.303480 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.303497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.303602 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.305130 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.305137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.305164 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.305620 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.305639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.307300 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.307305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.310121 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.310134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.312153 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.312161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.312338 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.312366 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.312446 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.312575 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"mutex_wait_us":243,"thread_start_us":1359,"threads_started":1}
I20260619 02:44:23.313913 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.301757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.7.246.67:34235)
0619 02:44:23.304727 (+ 2970us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.304739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.304778 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:23.304887 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.304898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.305147 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.305334 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.306050 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.307484 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.310282 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.311730 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.311868 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.312093 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.313399 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.313612 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2800,"thread_start_us":1656,"threads_started":1}
I20260619 02:44:23.316820 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa (DEFAULT_TABLE table=test-table [id=c3809906bd484eeba7871bac8d1f182d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.317004 10319 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa (DEFAULT_TABLE table=test-table [id=c3809906bd484eeba7871bac8d1f182d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.317008 10041 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa (DEFAULT_TABLE table=test-table [id=c3809906bd484eeba7871bac8d1f182d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.318173 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf054c7f2d4f108983e4387aa85aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.318347 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf054c7f2d4f108983e4387aa85aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.318363 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf054c7f2d4f108983e4387aa85aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.329897 10408 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Bootstrap starting.
I20260619 02:44:23.332554 10410 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Bootstrap starting.
I20260619 02:44:23.334255 10408 tablet_bootstrap.cc:654] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.334303 10409 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Bootstrap starting.
I20260619 02:44:23.335548 10408 log.cc:824] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:23.337844 10409 tablet_bootstrap.cc:654] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.338182 10408 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: No bootstrap required, opened a new log
I20260619 02:44:23.338454 10408 ts_tablet_manager.cc:1397] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 02:44:23.339109 10409 log.cc:824] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:23.339172 10410 tablet_bootstrap.cc:654] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.340353 10410 log.cc:824] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:23.342197 10409 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: No bootstrap required, opened a new log
I20260619 02:44:23.342469 10409 ts_tablet_manager.cc:1397] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 02:44:23.343667 10410 tablet_bootstrap.cc:492] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: No bootstrap required, opened a new log
I20260619 02:44:23.343930 10410 ts_tablet_manager.cc:1397] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 02:44:23.347144 10408 raft_consensus.cc:348] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.347493 10408 raft_consensus.cc:374] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.347589 10408 raft_consensus.cc:729] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.348223 10408 consensus_queue.cc:260] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.349396 10250 heartbeater.cc:503] Master 127.7.246.126:40137 was elected leader, sending a full tablet report...
I20260619 02:44:23.349920 10408 ts_tablet_manager.cc:1428] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 02:44:23.353606 10409 raft_consensus.cc:348] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.353587 10410 raft_consensus.cc:348] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.354044 10409 raft_consensus.cc:374] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.354043 10410 raft_consensus.cc:374] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.354140 10409 raft_consensus.cc:729] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4ecf34f0c49d78820302bc2d95b17, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.354142 10410 raft_consensus.cc:729] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.354724 10409 consensus_queue.cc:260] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.354830 10410 consensus_queue.cc:260] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.356619 10409 ts_tablet_manager.cc:1428] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260619 02:44:23.357661 10389 heartbeater.cc:503] Master 127.7.246.126:40137 was elected leader, sending a full tablet report...
I20260619 02:44:23.358278 10410 ts_tablet_manager.cc:1428] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260619 02:44:23.360615 10414 raft_consensus.cc:482] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:23.360882 10414 raft_consensus.cc:504] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.362615 10414 leader_election.cc:288] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235), 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407)
I20260619 02:44:23.369917 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.363057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:37969 (local address 127.7.246.65:33407)
0619 02:44:23.363214 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.363220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.363956 (+ 736us) server_negotiation.cc:407] Connection header received
0619 02:44:23.364131 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.364135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.364227 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.364356 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.365023 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.366170 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.368112 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.368919 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.369091 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.369176 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.369509 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.369664 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:44:23.369949 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.363024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:34879 (local address 127.7.246.67:34235)
0619 02:44:23.363209 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.363215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.364194 (+ 979us) server_negotiation.cc:407] Connection header received
0619 02:44:23.364331 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.364339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.364404 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.364485 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.365168 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.366199 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.368179 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.368916 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.369100 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.369188 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.369575 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.369703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:44:23.370116 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.363036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33407 (local address 127.7.246.66:37969)
0619 02:44:23.363813 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.363839 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.364009 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.364380 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.364383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.364406 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.364868 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.364884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.366298 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.366303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.367958 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.367973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.369158 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.369168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.369363 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.369386 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.369464 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.369662 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":7,"thread_start_us":389,"threads_started":1}
I20260619 02:44:23.370466 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.363036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:34235 (local address 127.7.246.66:34879)
0619 02:44:23.364106 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.364123 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.364223 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.364510 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.364520 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.364550 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.365034 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.365044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.366359 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.366363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.368001 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.368013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.369183 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.369192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.369378 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.369399 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.369509 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.369664 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":274,"thread_start_us":418,"threads_started":1}
I20260619 02:44:23.371105 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf054c7f2d4f108983e4387aa85aaa" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17" is_pre_election: true
I20260619 02:44:23.371270 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf054c7f2d4f108983e4387aa85aaa" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d" is_pre_election: true
I20260619 02:44:23.371695 10066 raft_consensus.cc:2454] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:23.371734 10339 raft_consensus.cc:2454] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:23.372313 10131 leader_election.cc:302] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:23.372706 10414 raft_consensus.cc:2781] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:23.372846 10414 raft_consensus.cc:482] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:23.372946 10414 raft_consensus.cc:3037] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.375125 10414 raft_consensus.cc:504] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.375707 10414 leader_election.cc:288] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Requested vote from peers 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235), 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407)
I20260619 02:44:23.376091 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf054c7f2d4f108983e4387aa85aaa" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d"
I20260619 02:44:23.376200 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf054c7f2d4f108983e4387aa85aaa" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17"
I20260619 02:44:23.376294 10339 raft_consensus.cc:3037] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.376401 10066 raft_consensus.cc:3037] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.379488 10066 raft_consensus.cc:2454] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:23.379488 10339 raft_consensus.cc:2454] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:23.379916 10131 leader_election.cc:302] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:23.380214 10414 raft_consensus.cc:2781] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:23.380647 10414 raft_consensus.cc:686] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Becoming Leader. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Running, Role: LEADER
I20260619 02:44:23.381104 10414 consensus_queue.cc:237] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } }
I20260619 02:44:23.385963 9903 catalog_manager.cc:5495] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:44:23 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837061, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:23.436180 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.423718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40137 (local address 127.0.0.1:47880)
0619 02:44:23.424233 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.424258 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.424375 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.424681 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.424684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.425336 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:23.425663 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.425672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.426759 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.426766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.427837 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.427848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.428571 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.428577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.428740 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.429336 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:23.429352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:23.431486 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:23.434121 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.434131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.434137 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.434528 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.434831 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:23.434835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:23.434837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:23.434958 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:23.435408 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:23.435414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:23.435541 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.435675 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.435854 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260619 02:44:23.436198 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.423702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.7.246.126:40137)
0619 02:44:23.423866 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.423871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.424340 (+ 469us) server_negotiation.cc:407] Connection header received
0619 02:44:23.424479 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.424483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.424550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.424660 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:23.425818 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.426536 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.427973 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.428390 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.428531 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.431671 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:23.431675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:23.431679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:23.431705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:23.434005 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.434623 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.434626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.434629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.434741 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:23.435044 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:23.435047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:23.435049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:23.435300 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:23.435397 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.435759 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.435969 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":59}
I20260619 02:44:23.441138 9903 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47880:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:44:23.442168 9903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260619 02:44:23.445693 10112 tablet.cc:2369] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:23.448659 10041 tablet_service.cc:1467] Processing CreateTablet for tablet def2152b4c014671a26e3478dd5e3a17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16b72bd2f65c4f3e86369a19b889ab6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:23.448882 10177 tablet_service.cc:1467] Processing CreateTablet for tablet def2152b4c014671a26e3478dd5e3a17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16b72bd2f65c4f3e86369a19b889ab6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:23.449848 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def2152b4c014671a26e3478dd5e3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.449965 10319 tablet_service.cc:1467] Processing CreateTablet for tablet def2152b4c014671a26e3478dd5e3a17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16b72bd2f65c4f3e86369a19b889ab6b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:44:23.450628 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def2152b4c014671a26e3478dd5e3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.455809 10408 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: Bootstrap starting.
I20260619 02:44:23.455905 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def2152b4c014671a26e3478dd5e3a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.456218 10410 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: Bootstrap starting.
I20260619 02:44:23.457499 10408 tablet_bootstrap.cc:654] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.457795 10410 tablet_bootstrap.cc:654] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: Neither blocks nor log segments found. Creating new log.
W20260619 02:44:23.458315 10251 tablet.cc:2369] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:23.461938 10409 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: Bootstrap starting.
I20260619 02:44:23.463654 10409 tablet_bootstrap.cc:654] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.463668 10408 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: No bootstrap required, opened a new log
I20260619 02:44:23.463687 10410 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: No bootstrap required, opened a new log
I20260619 02:44:23.463812 10408 ts_tablet_manager.cc:1397] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 02:44:23.463840 10410 ts_tablet_manager.cc:1397] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260619 02:44:23.464609 10410 raft_consensus.cc:348] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.464818 10410 raft_consensus.cc:374] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.464630 10408 raft_consensus.cc:348] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.464912 10410 raft_consensus.cc:729] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.464951 10408 raft_consensus.cc:374] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.465060 10408 raft_consensus.cc:729] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.465167 10410 consensus_queue.cc:260] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.465294 10408 consensus_queue.cc:260] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.465691 10414 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.465682 10416 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.465952 10408 ts_tablet_manager.cc:1428] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:44:23.466042 10416 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.466434 10410 ts_tablet_manager.cc:1428] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:44:23.465978 10414 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.467669 10409 tablet_bootstrap.cc:492] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: No bootstrap required, opened a new log
I20260619 02:44:23.467823 10409 ts_tablet_manager.cc:1397] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 02:44:23.468552 10409 raft_consensus.cc:348] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.468817 10409 raft_consensus.cc:374] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.468907 10409 raft_consensus.cc:729] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4ecf34f0c49d78820302bc2d95b17, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.469129 10409 consensus_queue.cc:260] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.469859 10409 ts_tablet_manager.cc:1428] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 02:44:23.469698 10415 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.470077 10415 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 02:44:23.475126 10390 tablet.cc:2369] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:23.505951 10414 raft_consensus.cc:482] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:23.506156 10414 raft_consensus.cc:504] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.506839 10414 leader_election.cc:288] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:23.507314 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def2152b4c014671a26e3478dd5e3a17" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17" is_pre_election: true
I20260619 02:44:23.507452 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def2152b4c014671a26e3478dd5e3a17" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d" is_pre_election: true
I20260619 02:44:23.507606 10066 raft_consensus.cc:2454] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:23.507694 10339 raft_consensus.cc:2454] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:23.508021 10133 leader_election.cc:302] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c4ecf34f0c49d78820302bc2d95b17, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:23.508365 10414 raft_consensus.cc:2781] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:23.508459 10414 raft_consensus.cc:482] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:23.508546 10414 raft_consensus.cc:3037] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.510681 10414 raft_consensus.cc:504] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.511229 10414 leader_election.cc:288] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Requested vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:23.511556 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def2152b4c014671a26e3478dd5e3a17" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17"
I20260619 02:44:23.511663 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def2152b4c014671a26e3478dd5e3a17" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d"
I20260619 02:44:23.511797 10066 raft_consensus.cc:3037] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.511932 10339 raft_consensus.cc:3037] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.513880 10066 raft_consensus.cc:2454] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:23.513878 10339 raft_consensus.cc:2454] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:23.514267 10133 leader_election.cc:302] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c4ecf34f0c49d78820302bc2d95b17, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:23.514509 10414 raft_consensus.cc:2781] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:23.514724 10414 raft_consensus.cc:686] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Becoming Leader. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Running, Role: LEADER
I20260619 02:44:23.514987 10414 consensus_queue.cc:237] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.516491 10419 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: TxnStatusTablet state changed. Reason: New leader a0f37f6871b645a28dc31c4f266817c9. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.516812 10419 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:23.517607 10431 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:44:23.517925 10431 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:44:23.518133 9903 catalog_manager.cc:5495] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:23.544049 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.538250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.7.246.66:42825)
0619 02:44:23.538459 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.538465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.538615 (+ 150us) server_negotiation.cc:407] Connection header received
0619 02:44:23.538908 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.538915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.538989 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.539125 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.539480 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.540346 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.541764 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.542225 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.542385 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.542892 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.543438 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.543605 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.543660 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.543745 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:44:23.544078 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.538256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.0.0.1:43962)
0619 02:44:23.538533 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.538548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.538704 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.539145 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.539149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.539161 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:23.539344 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.539352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.540463 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.540470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.541627 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.541641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.542403 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.542409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.542631 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.542688 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.543549 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.543568 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.543643 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.543798 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 02:44:23.552143 10339 raft_consensus.cc:1261] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.552135 10066 raft_consensus.cc:1261] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.553128 10419 consensus_queue.cc:1035] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.553683 10414 consensus_queue.cc:1035] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.566896 10416 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: TxnStatusTablet state changed. Reason: New leader a0f37f6871b645a28dc31c4f266817c9. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.567193 10416 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.569898 10434 mvcc.cc:205] Tried to move back new op lower bound from 7298404612299223040 to 7298404612159524864. Current Snapshot: MvccSnapshot[applied={T|T < 7298404612299223040}]
I20260619 02:44:23.572072 10414 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.572387 10414 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:23.575344 10415 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: TxnStatusTablet state changed. Reason: New leader a0f37f6871b645a28dc31c4f266817c9. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.575620 10415 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.578011 10416 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.578227 10416 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 01615a0d99bd4febb644911cedf39f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.580876 10414 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.581139 10414 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P a0f37f6871b645a28dc31c4f266817c9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:44:23.581698 10415 tablet_replica.cc:439] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } } }
I20260619 02:44:23.581913 10415 tablet_replica.cc:442] T def2152b4c014671a26e3478dd5e3a17 P 80c4ecf34f0c49d78820302bc2d95b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:44:23.589018 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.584374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.7.246.66:42825)
0619 02:44:23.584534 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.584538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.584554 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:44:23.584598 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.584601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.584651 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.584758 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.585112 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.586063 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.587334 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.587740 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.587874 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.588295 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.588410 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.588512 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.588660 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.588739 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:44:23.605428 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.599851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:39363 (local address 127.7.246.66:42825)
0619 02:44:23.600029 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.600035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.600203 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:44:23.600462 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.600473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.600525 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.600608 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.601141 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.601942 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.603502 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.604090 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.604232 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.604660 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.604840 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.604934 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.605060 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.605123 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":66}
I20260619 02:44:23.605772 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.599515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.7.246.66:39363)
0619 02:44:23.600110 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.600131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.600340 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.600628 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.600632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.600642 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:23.600994 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.601005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.602082 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.602086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.603365 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.603376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.604278 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.604287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.604421 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.604475 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.604931 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.604940 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.605035 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.605176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":3,"thread_start_us":537,"threads_started":1}
I20260619 02:44:23.616251 10065 raft_consensus.cc:1261] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.616410 10339 raft_consensus.cc:1261] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.616907 10441 consensus_queue.cc:1035] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.617280 10414 consensus_queue.cc:1035] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.639616 10436 mvcc.cc:205] Tried to move back new op lower bound from 7298404612567031808 to 7298404611613466624. Current Snapshot: MvccSnapshot[applied={T|T < 7298404612567031808}]
I20260619 02:44:23.690246 9903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:23.690543 9903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:23.694154 9903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Sending DeleteTablet for 3 replicas of tablet 1ebf054c7f2d4f108983e4387aa85aaa
I20260619 02:44:23.695442 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260619 02:44:23.695449 10180 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260619 02:44:23.695467 10319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebf054c7f2d4f108983e4387aa85aaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35814
I20260619 02:44:23.697872 9903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:23.699580 9903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:23.706169 10453 tablet_replica.cc:330] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: stopping tablet replica
I20260619 02:44:23.707365 10453 raft_consensus.cc:2229] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:23.708447 10453 raft_consensus.cc:2258] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:23.710765 10452 tablet_replica.cc:330] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: stopping tablet replica
I20260619 02:44:23.711225 10452 raft_consensus.cc:2229] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:23.711409 10453 ts_tablet_manager.cc:1905] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:23.711565 10452 raft_consensus.cc:2258] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:23.715237 10452 ts_tablet_manager.cc:1905] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:23.718950 10040 tablet_service.cc:1467] Processing CreateTablet for tablet 4f98d8e6df9048f796636ee438db995f (DEFAULT_TABLE table=test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.719314 10319 tablet_service.cc:1467] Processing CreateTablet for tablet 4f98d8e6df9048f796636ee438db995f (DEFAULT_TABLE table=test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.719522 10040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f98d8e6df9048f796636ee438db995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.719937 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f98d8e6df9048f796636ee438db995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.722988 10180 tablet_service.cc:1467] Processing CreateTablet for tablet 4f98d8e6df9048f796636ee438db995f (DEFAULT_TABLE table=test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:23.723498 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f98d8e6df9048f796636ee438db995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:23.725332 10453 ts_tablet_manager.cc:1918] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:23.725533 10453 log.cc:1196] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/1ebf054c7f2d4f108983e4387aa85aaa
I20260619 02:44:23.726563 10453 ts_tablet_manager.cc:1939] T 1ebf054c7f2d4f108983e4387aa85aaa P a0f37f6871b645a28dc31c4f266817c9: Deleting consensus metadata
I20260619 02:44:23.728631 9890 catalog_manager.cc:4841] TS a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825): tablet 1ebf054c7f2d4f108983e4387aa85aaa (table test-table [id=c3809906bd484eeba7871bac8d1f182d]) successfully deleted
I20260619 02:44:23.731283 10409 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Bootstrap starting.
I20260619 02:44:23.731810 10410 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Bootstrap starting.
I20260619 02:44:23.732992 10409 tablet_bootstrap.cc:654] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.733482 10410 tablet_bootstrap.cc:654] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.734867 10408 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Bootstrap starting.
I20260619 02:44:23.736937 10408 tablet_bootstrap.cc:654] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:23.742493 10409 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: No bootstrap required, opened a new log
I20260619 02:44:23.742709 10409 ts_tablet_manager.cc:1397] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260619 02:44:23.743577 10451 tablet_replica.cc:330] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: stopping tablet replica
I20260619 02:44:23.743636 10409 raft_consensus.cc:348] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.744432 10409 raft_consensus.cc:374] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.744568 10409 raft_consensus.cc:729] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4ecf34f0c49d78820302bc2d95b17, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.744899 10409 consensus_queue.cc:260] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.745605 10409 ts_tablet_manager.cc:1428] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 02:44:23.746114 10451 raft_consensus.cc:2229] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:23.746495 10451 raft_consensus.cc:2258] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:23.749509 10408 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: No bootstrap required, opened a new log
I20260619 02:44:23.750048 10408 ts_tablet_manager.cc:1397] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20260619 02:44:23.750066 10410 tablet_bootstrap.cc:492] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: No bootstrap required, opened a new log
I20260619 02:44:23.750238 10410 ts_tablet_manager.cc:1397] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260619 02:44:23.750910 10408 raft_consensus.cc:348] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.751128 10408 raft_consensus.cc:374] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.751063 10410 raft_consensus.cc:348] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.751231 10408 raft_consensus.cc:729] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.751327 10410 raft_consensus.cc:374] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:23.751412 10410 raft_consensus.cc:729] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Initialized, Role: FOLLOWER
I20260619 02:44:23.751519 10408 consensus_queue.cc:260] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.751657 10410 consensus_queue.cc:260] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.752584 10408 ts_tablet_manager.cc:1428] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:44:23.757019 10410 ts_tablet_manager.cc:1428] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260619 02:44:23.757082 10451 ts_tablet_manager.cc:1905] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:23.758755 10452 ts_tablet_manager.cc:1918] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:23.758924 10452 log.cc:1196] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/1ebf054c7f2d4f108983e4387aa85aaa
I20260619 02:44:23.759387 10452 ts_tablet_manager.cc:1939] T 1ebf054c7f2d4f108983e4387aa85aaa P 01615a0d99bd4febb644911cedf39f3d: Deleting consensus metadata
I20260619 02:44:23.761010 9888 catalog_manager.cc:4841] TS 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235): tablet 1ebf054c7f2d4f108983e4387aa85aaa (table test-table [id=c3809906bd484eeba7871bac8d1f182d]) successfully deleted
I20260619 02:44:23.765875 10451 ts_tablet_manager.cc:1918] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:23.766036 10451 log.cc:1196] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/1ebf054c7f2d4f108983e4387aa85aaa
I20260619 02:44:23.766531 10451 ts_tablet_manager.cc:1939] T 1ebf054c7f2d4f108983e4387aa85aaa P 80c4ecf34f0c49d78820302bc2d95b17: Deleting consensus metadata
I20260619 02:44:23.768129 9890 catalog_manager.cc:4841] TS 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407): tablet 1ebf054c7f2d4f108983e4387aa85aaa (table test-table [id=c3809906bd484eeba7871bac8d1f182d]) successfully deleted
I20260619 02:44:23.782284 10416 raft_consensus.cc:482] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:23.782552 10416 raft_consensus.cc:504] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.784408 10416 leader_election.cc:288] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825)
I20260619 02:44:23.792176 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.784718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:38647 (local address 127.7.246.65:33407)
0619 02:44:23.784940 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.784947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.785494 (+ 547us) server_negotiation.cc:407] Connection header received
0619 02:44:23.785814 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.785820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.785899 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.785962 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.786678 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.787436 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.790118 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.791162 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.791336 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.791405 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.791784 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.791907 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:44:23.792210 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.784619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33407 (local address 127.7.246.67:38647)
0619 02:44:23.785387 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.785408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.785632 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.786211 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.786216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.786247 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.786514 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.786528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.787567 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.787573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.789950 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.789966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.791496 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.791500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.791622 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.791644 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.791752 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.791879 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":458,"threads_started":1}
I20260619 02:44:23.792861 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f98d8e6df9048f796636ee438db995f" candidate_uuid: "01615a0d99bd4febb644911cedf39f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17" is_pre_election: true
I20260619 02:44:23.793154 10066 raft_consensus.cc:2454] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01615a0d99bd4febb644911cedf39f3d in term 0.
I20260619 02:44:23.793634 10272 leader_election.cc:302] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, 80c4ecf34f0c49d78820302bc2d95b17; no voters:
I20260619 02:44:23.793938 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.785535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:59917 (local address 127.7.246.66:42825)
0619 02:44:23.785763 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.785772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.786296 (+ 524us) server_negotiation.cc:407] Connection header received
0619 02:44:23.786505 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.786509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.786562 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.786663 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:23.788083 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.788791 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.790638 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.792177 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.792920 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.793010 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.793482 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.793652 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:44:23.794359 10416 raft_consensus.cc:2781] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:23.794510 10416 raft_consensus.cc:482] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:23.794552 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.785466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.7.246.67:59917)
0619 02:44:23.786184 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.786201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.786363 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.787576 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.787579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.787600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:23.787907 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.787918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.788934 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.788942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.790480 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.790494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.792960 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.792967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.793151 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.793189 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.793995 (+ 806us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.794127 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"spinlock_wait_cycles":1280,"thread_start_us":467,"threads_started":1}
I20260619 02:44:23.794620 10416 raft_consensus.cc:3037] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.795398 10205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f98d8e6df9048f796636ee438db995f" candidate_uuid: "01615a0d99bd4febb644911cedf39f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f37f6871b645a28dc31c4f266817c9" is_pre_election: true
I20260619 02:44:23.795828 10205 raft_consensus.cc:2454] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01615a0d99bd4febb644911cedf39f3d in term 0.
I20260619 02:44:23.797837 10416 raft_consensus.cc:504] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.798544 10416 leader_election.cc:288] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [CANDIDATE]: Term 1 election: Requested vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825)
I20260619 02:44:23.799012 10205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f98d8e6df9048f796636ee438db995f" candidate_uuid: "01615a0d99bd4febb644911cedf39f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f37f6871b645a28dc31c4f266817c9"
I20260619 02:44:23.798944 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f98d8e6df9048f796636ee438db995f" candidate_uuid: "01615a0d99bd4febb644911cedf39f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17"
I20260619 02:44:23.799201 10205 raft_consensus.cc:3037] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.799214 10066 raft_consensus.cc:3037] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:23.801357 10205 raft_consensus.cc:2454] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01615a0d99bd4febb644911cedf39f3d in term 1.
I20260619 02:44:23.801744 10272 leader_election.cc:302] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:23.802045 10416 raft_consensus.cc:2781] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:23.802249 10066 raft_consensus.cc:2454] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01615a0d99bd4febb644911cedf39f3d in term 1.
I20260619 02:44:23.802551 10416 raft_consensus.cc:686] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 1 LEADER]: Becoming Leader. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Running, Role: LEADER
I20260619 02:44:23.803009 10416 consensus_queue.cc:237] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:23.806473 9904 catalog_manager.cc:5495] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 01615a0d99bd4febb644911cedf39f3d (127.7.246.67). New cstate: current_term: 1 leader_uuid: "01615a0d99bd4febb644911cedf39f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } health_report { overall_health: HEALTHY } } }
I20260619 02:44:23.842249 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.836029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.7.246.67:34235)
0619 02:44:23.836228 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.836235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.836259 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:44:23.836440 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.836446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.836588 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.836775 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.837092 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.838302 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.839567 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.840235 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.840380 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.840790 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.841593 (+ 803us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.841748 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.841800 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.841888 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:44:23.859251 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.853283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:57017 (local address 127.7.246.66:42825)
0619 02:44:23.853530 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.853539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.854159 (+ 620us) server_negotiation.cc:407] Connection header received
0619 02:44:23.854375 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.854382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.854448 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.854558 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.855154 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.855947 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.857302 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.857821 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.857968 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.858437 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.858556 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.858681 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.858820 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.858900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:44:23.859287 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.853196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.7.246.67:57017)
0619 02:44:23.854055 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.854072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.854228 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.854575 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.854580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.854592 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:23.854971 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.854981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.856095 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.856103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.857162 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.857172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.858002 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.858008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.858157 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.858256 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.858681 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.858701 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.858796 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.858926 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":566,"threads_started":1}
I20260619 02:44:23.868554 10066 raft_consensus.cc:1261] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Refusing update from remote peer 01615a0d99bd4febb644911cedf39f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.868660 10205 raft_consensus.cc:1261] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Refusing update from remote peer 01615a0d99bd4febb644911cedf39f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:23.869344 10459 consensus_queue.cc:1035] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.869804 10416 consensus_queue.cc:1035] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:23.901378 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.895367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:34235 (local address 127.7.246.66:37913)
0619 02:44:23.895553 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:23.895583 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:23.895691 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:23.896249 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:23.896254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:23.896267 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:44:23.896494 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.896507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.897577 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.897583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.899325 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:23.899345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.900105 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.900112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:23.900337 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.900411 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.900855 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.900868 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 02:44:23.900949 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:44:23.901117 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:44:23.902129 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:23.895329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:37913 (local address 127.7.246.67:34235)
0619 02:44:23.895626 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:23.895632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:23.895654 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:44:23.895832 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:23.895851 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:23.895953 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:23.896064 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:23.896629 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.897407 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.899498 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:23.899942 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:23.900038 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:23.900634 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:23.900763 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:23.900852 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:23.901727 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 02:44:23.901834 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
W20260619 02:44:23.903390 10435 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:44:25.496495 9904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:25.496730 9904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:25.500844 9904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Sending DeleteTablet for 3 replicas of tablet 4f98d8e6df9048f796636ee438db995f
I20260619 02:44:25.501621 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f98d8e6df9048f796636ee438db995f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260619 02:44:25.501700 10319 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f98d8e6df9048f796636ee438db995f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35814
I20260619 02:44:25.501678 10180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f98d8e6df9048f796636ee438db995f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260619 02:44:25.516994 10492 tablet_replica.cc:330] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: stopping tablet replica
I20260619 02:44:25.517256 10492 raft_consensus.cc:2229] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:25.517530 10492 raft_consensus.cc:2258] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.518496 10494 tablet_replica.cc:330] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: stopping tablet replica
I20260619 02:44:25.518738 10494 raft_consensus.cc:2229] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:25.519055 10492 ts_tablet_manager.cc:1905] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.519330 10493 tablet_replica.cc:330] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: stopping tablet replica
I20260619 02:44:25.519553 10493 raft_consensus.cc:2229] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:25.519804 10493 raft_consensus.cc:2258] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.520144 10494 raft_consensus.cc:2258] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.520952 10493 ts_tablet_manager.cc:1905] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.522169 10494 ts_tablet_manager.cc:1905] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.525283 10492 ts_tablet_manager.cc:1918] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:25.525425 10492 log.cc:1196] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/4f98d8e6df9048f796636ee438db995f
I20260619 02:44:25.525892 10492 ts_tablet_manager.cc:1939] T 4f98d8e6df9048f796636ee438db995f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting consensus metadata
I20260619 02:44:25.527055 9890 catalog_manager.cc:4841] TS 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407): tablet 4f98d8e6df9048f796636ee438db995f (table test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]) successfully deleted
I20260619 02:44:25.528131 10493 ts_tablet_manager.cc:1918] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:25.528299 10493 log.cc:1196] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/4f98d8e6df9048f796636ee438db995f
I20260619 02:44:25.528720 10493 ts_tablet_manager.cc:1939] T 4f98d8e6df9048f796636ee438db995f P a0f37f6871b645a28dc31c4f266817c9: Deleting consensus metadata
I20260619 02:44:25.528719 10494 ts_tablet_manager.cc:1918] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:25.528848 10494 log.cc:1196] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/4f98d8e6df9048f796636ee438db995f
I20260619 02:44:25.529287 10494 ts_tablet_manager.cc:1939] T 4f98d8e6df9048f796636ee438db995f P 01615a0d99bd4febb644911cedf39f3d: Deleting consensus metadata
I20260619 02:44:25.529737 9890 catalog_manager.cc:4841] TS a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825): tablet 4f98d8e6df9048f796636ee438db995f (table test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]) successfully deleted
I20260619 02:44:25.530520 9888 catalog_manager.cc:4841] TS 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235): tablet 4f98d8e6df9048f796636ee438db995f (table test-table [id=9f6f9c1ce7f94c92b31cc4a725155a9b]) successfully deleted
Jun 19 02:44:25 dist-test-slave-9tkz krb5kdc[9859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:44:25 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:25.563434 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.549077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.7.246.126:40137)
0619 02:44:25.549699 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:25.549703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:25.549725 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:44:25.549943 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:25.549947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:25.550009 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:25.550111 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:25.551428 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.552182 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.553444 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.553888 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.553978 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.558441 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:25.558446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:25.558450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:25.558483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:25.561024 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:25.561787 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:25.561791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:25.561793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:25.561917 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:25.562258 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:25.562265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:25.562269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:25.562523 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:25.562732 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:25.563019 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:44:25.563173 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":304,"threads_started":1}
I20260619 02:44:25.566757 9904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:25.567386 9904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:25.573676 10041 tablet_service.cc:1467] Processing CreateTablet for tablet a528b0860a4b445d99640a664447fe6f (DEFAULT_TABLE table=test-table [id=01c929d378104ee8b23bcbd181e935aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.573859 10180 tablet_service.cc:1467] Processing CreateTablet for tablet a528b0860a4b445d99640a664447fe6f (DEFAULT_TABLE table=test-table [id=01c929d378104ee8b23bcbd181e935aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.573939 10319 tablet_service.cc:1467] Processing CreateTablet for tablet a528b0860a4b445d99640a664447fe6f (DEFAULT_TABLE table=test-table [id=01c929d378104ee8b23bcbd181e935aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.574133 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528b0860a4b445d99640a664447fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.574270 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528b0860a4b445d99640a664447fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.574316 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a528b0860a4b445d99640a664447fe6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.580957 10504 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Bootstrap starting.
I20260619 02:44:25.581926 10503 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Bootstrap starting.
I20260619 02:44:25.582659 10505 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Bootstrap starting.
I20260619 02:44:25.583614 10503 tablet_bootstrap.cc:654] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.584182 10504 tablet_bootstrap.cc:654] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.584373 10505 tablet_bootstrap.cc:654] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.585626 10503 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: No bootstrap required, opened a new log
I20260619 02:44:25.585775 10503 ts_tablet_manager.cc:1397] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:44:25.586181 10504 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: No bootstrap required, opened a new log
I20260619 02:44:25.586318 10504 ts_tablet_manager.cc:1397] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:44:25.586444 10503 raft_consensus.cc:348] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.587141 10503 raft_consensus.cc:374] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.587240 10503 raft_consensus.cc:729] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.587056 10504 raft_consensus.cc:348] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.587370 10504 raft_consensus.cc:374] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.587486 10504 raft_consensus.cc:729] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4ecf34f0c49d78820302bc2d95b17, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.587502 10503 consensus_queue.cc:260] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.587837 10504 consensus_queue.cc:260] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.588757 10504 ts_tablet_manager.cc:1428] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:44:25.589202 10503 ts_tablet_manager.cc:1428] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:25.590022 10505 tablet_bootstrap.cc:492] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: No bootstrap required, opened a new log
I20260619 02:44:25.590185 10505 ts_tablet_manager.cc:1397] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 02:44:25.590955 10505 raft_consensus.cc:348] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.591243 10505 raft_consensus.cc:374] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.591352 10505 raft_consensus.cc:729] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.591662 10505 consensus_queue.cc:260] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.592299 10505 ts_tablet_manager.cc:1428] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:44:25.779596 10509 raft_consensus.cc:482] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:25.779866 10509 raft_consensus.cc:504] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.781405 10509 leader_election.cc:288] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:25.798993 10469 raft_consensus.cc:482] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:25.799468 10469 raft_consensus.cc:504] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.800268 10469 leader_election.cc:288] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:25.800768 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17" is_pre_election: true
I20260619 02:44:25.801003 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d" is_pre_election: true
I20260619 02:44:25.801136 10066 raft_consensus.cc:2454] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:25.801245 10339 raft_consensus.cc:2454] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:25.801678 10133 leader_election.cc:302] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c4ecf34f0c49d78820302bc2d95b17, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:25.801973 10469 raft_consensus.cc:2781] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:25.802083 10469 raft_consensus.cc:482] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:25.802172 10469 raft_consensus.cc:3037] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.804955 10469 raft_consensus.cc:504] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.805513 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.781854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:57197 (local address 127.7.246.67:34235)
0619 02:44:25.797048 (+ 15194us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:25.797058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:25.798181 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 02:44:25.798397 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:25.798401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:25.798455 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:25.798557 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:25.799621 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.800560 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.803092 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.804392 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.804553 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.804631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:25.805021 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 02:44:25.805166 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14959,"thread_start_us":14731,"threads_started":1}
I20260619 02:44:25.805572 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.781860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:34235 (local address 127.7.246.65:57197)
0619 02:44:25.798071 (+ 16211us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:25.798084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:25.798243 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:25.798579 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:25.798584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:25.798609 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:25.799469 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:25.799480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.800725 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.800729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:25.802933 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:25.802947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.804583 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.804590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:25.804831 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.804875 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:44:25.804994 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:44:25.805146 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16115,"spinlock_wait_cycles":5888,"thread_start_us":15265,"threads_started":1}
I20260619 02:44:25.806151 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "80c4ecf34f0c49d78820302bc2d95b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d" is_pre_election: true
I20260619 02:44:25.806286 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.781854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:37707 (local address 127.7.246.66:42825)
0619 02:44:25.797906 (+ 16052us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:25.797912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:25.797935 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:44:25.798012 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:25.798016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:25.798072 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:25.798230 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:25.798802 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.800236 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.803271 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.804616 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.804796 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.804867 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:25.805793 (+ 926us) server_negotiation.cc:299] Negotiation successful
0619 02:44:25.805930 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15904,"thread_start_us":15747,"threads_started":1}
I20260619 02:44:25.805613 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.781809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:42825 (local address 127.7.246.65:37707)
0619 02:44:25.797355 (+ 15546us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:25.797384 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:25.797618 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:25.798249 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:25.798254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:25.798305 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:25.798597 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:25.798607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.800422 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.800428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:25.803110 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:25.803124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.804803 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.804809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:25.805036 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.805067 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:44:25.805150 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:44:25.805278 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15350,"thread_start_us":16007,"threads_started":1}
I20260619 02:44:25.806401 10339 raft_consensus.cc:2454] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c4ecf34f0c49d78820302bc2d95b17 in term 0.
I20260619 02:44:25.806301 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17"
I20260619 02:44:25.806562 10066 raft_consensus.cc:3037] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.806651 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d"
I20260619 02:44:25.806876 10338 raft_consensus.cc:3037] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.807008 9991 leader_election.cc:302] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, 80c4ecf34f0c49d78820302bc2d95b17; no voters:
I20260619 02:44:25.807436 10469 leader_election.cc:288] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Requested vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:25.809178 10066 raft_consensus.cc:2454] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:25.809358 10509 raft_consensus.cc:2741] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 02:44:25.809670 10338 raft_consensus.cc:2454] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:25.810086 10131 leader_election.cc:302] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01615a0d99bd4febb644911cedf39f3d, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:25.810115 10204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a528b0860a4b445d99640a664447fe6f" candidate_uuid: "80c4ecf34f0c49d78820302bc2d95b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0f37f6871b645a28dc31c4f266817c9" is_pre_election: true
I20260619 02:44:25.810423 10204 raft_consensus.cc:2379] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 80c4ecf34f0c49d78820302bc2d95b17 in current term 1: Already voted for candidate a0f37f6871b645a28dc31c4f266817c9 in this term.
I20260619 02:44:25.810624 10469 raft_consensus.cc:2781] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:25.810798 10469 raft_consensus.cc:686] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Becoming Leader. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Running, Role: LEADER
I20260619 02:44:25.811062 10469 consensus_queue.cc:237] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.813772 9902 catalog_manager.cc:5495] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:25.883898 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:25.879997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.7.246.66:42825)
0619 02:44:25.880168 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:25.880172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:25.880187 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:44:25.880304 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:25.880307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:25.880371 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:25.880442 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:25.880737 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.881413 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.882623 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:25.882982 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:25.883070 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:25.883395 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:25.883481 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:25.883542 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:25.883657 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:44:25.883711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:44:25.892153 10338 raft_consensus.cc:1261] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:25.892632 10066 raft_consensus.cc:1261] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:25.892611 10469 consensus_queue.cc:1035] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:25.893111 10469 consensus_queue.cc:1035] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:25.933284 9902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:25.933522 9902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:25.936218 9902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Sending DeleteTablet for 3 replicas of tablet a528b0860a4b445d99640a664447fe6f
I20260619 02:44:25.937119 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet a528b0860a4b445d99640a664447fe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260619 02:44:25.937184 10319 tablet_service.cc:1514] Processing DeleteTablet for tablet a528b0860a4b445d99640a664447fe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35814
I20260619 02:44:25.937397 10492 tablet_replica.cc:330] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: stopping tablet replica
I20260619 02:44:25.937642 10492 raft_consensus.cc:2229] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:25.937701 10494 tablet_replica.cc:330] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: stopping tablet replica
I20260619 02:44:25.937849 10492 raft_consensus.cc:2258] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.937906 10494 raft_consensus.cc:2229] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:25.938105 10494 raft_consensus.cc:2258] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.938169 9902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:44:25.938424 10180 tablet_service.cc:1514] Processing DeleteTablet for tablet a528b0860a4b445d99640a664447fe6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:25 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:43958
W20260619 02:44:25.938838 9902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:25.938890 10492 ts_tablet_manager.cc:1905] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.939144 10494 ts_tablet_manager.cc:1905] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.944299 10493 tablet_replica.cc:330] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: stopping tablet replica
I20260619 02:44:25.944537 10493 raft_consensus.cc:2229] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:25.944847 10493 raft_consensus.cc:2258] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:25.945446 10494 ts_tablet_manager.cc:1918] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:25.945631 10494 log.cc:1196] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/a528b0860a4b445d99640a664447fe6f
I20260619 02:44:25.945796 10041 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea92544abe4dac92796625580c1f4e (DEFAULT_TABLE table=test-table [id=04b485702e11435b9399e639a4b0235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.946035 10493 ts_tablet_manager.cc:1905] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:25.946101 10494 ts_tablet_manager.cc:1939] T a528b0860a4b445d99640a664447fe6f P 01615a0d99bd4febb644911cedf39f3d: Deleting consensus metadata
I20260619 02:44:25.946204 10041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea92544abe4dac92796625580c1f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.947336 10492 ts_tablet_manager.cc:1918] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:25.947512 10492 log.cc:1196] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/a528b0860a4b445d99640a664447fe6f
I20260619 02:44:25.947914 10492 ts_tablet_manager.cc:1939] T a528b0860a4b445d99640a664447fe6f P 80c4ecf34f0c49d78820302bc2d95b17: Deleting consensus metadata
I20260619 02:44:25.948853 9890 catalog_manager.cc:4841] TS 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407): tablet a528b0860a4b445d99640a664447fe6f (table test-table [id=01c929d378104ee8b23bcbd181e935aa]) successfully deleted
I20260619 02:44:25.952909 10504 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Bootstrap starting.
I20260619 02:44:25.953155 10180 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea92544abe4dac92796625580c1f4e (DEFAULT_TABLE table=test-table [id=04b485702e11435b9399e639a4b0235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.953424 10493 ts_tablet_manager.cc:1918] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:44:25.953593 10493 log.cc:1196] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/a528b0860a4b445d99640a664447fe6f
I20260619 02:44:25.953851 9888 catalog_manager.cc:4841] TS 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235): tablet a528b0860a4b445d99640a664447fe6f (table test-table [id=01c929d378104ee8b23bcbd181e935aa]) successfully deleted
I20260619 02:44:25.953608 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea92544abe4dac92796625580c1f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.954078 10493 ts_tablet_manager.cc:1939] T a528b0860a4b445d99640a664447fe6f P a0f37f6871b645a28dc31c4f266817c9: Deleting consensus metadata
I20260619 02:44:25.954169 10504 tablet_bootstrap.cc:654] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.955009 9890 catalog_manager.cc:4841] TS a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825): tablet a528b0860a4b445d99640a664447fe6f (table test-table [id=01c929d378104ee8b23bcbd181e935aa]) successfully deleted
I20260619 02:44:25.955614 10319 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea92544abe4dac92796625580c1f4e (DEFAULT_TABLE table=test-table [id=04b485702e11435b9399e639a4b0235a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:25.956030 10319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea92544abe4dac92796625580c1f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:25.956184 10504 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: No bootstrap required, opened a new log
I20260619 02:44:25.956303 10504 ts_tablet_manager.cc:1397] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:25.956796 10504 raft_consensus.cc:348] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.956964 10504 raft_consensus.cc:374] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.957007 10504 raft_consensus.cc:729] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c4ecf34f0c49d78820302bc2d95b17, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.957221 10504 consensus_queue.cc:260] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.957701 10504 ts_tablet_manager.cc:1428] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 02:44:25.962330 10503 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Bootstrap starting.
I20260619 02:44:25.962512 10505 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Bootstrap starting.
I20260619 02:44:25.963971 10505 tablet_bootstrap.cc:654] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.963963 10503 tablet_bootstrap.cc:654] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:25.965896 10505 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: No bootstrap required, opened a new log
I20260619 02:44:25.966056 10505 ts_tablet_manager.cc:1397] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:44:25.966367 10503 tablet_bootstrap.cc:492] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: No bootstrap required, opened a new log
I20260619 02:44:25.966513 10503 ts_tablet_manager.cc:1397] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:44:25.966823 10505 raft_consensus.cc:348] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.967072 10505 raft_consensus.cc:374] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.967157 10505 raft_consensus.cc:729] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01615a0d99bd4febb644911cedf39f3d, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.967206 10503 raft_consensus.cc:348] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.967445 10503 raft_consensus.cc:374] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:25.967518 10503 raft_consensus.cc:729] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Initialized, Role: FOLLOWER
I20260619 02:44:25.967417 10505 consensus_queue.cc:260] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.967731 10503 consensus_queue.cc:260] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.967932 10505 ts_tablet_manager.cc:1428] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 02:44:25.968247 10503 ts_tablet_manager.cc:1428] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:44:25.987113 10473 raft_consensus.cc:482] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:25.987324 10473 raft_consensus.cc:504] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.988085 10473 leader_election.cc:288] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:25.988549 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea92544abe4dac92796625580c1f4e" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17" is_pre_election: true
I20260619 02:44:25.988620 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea92544abe4dac92796625580c1f4e" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d" is_pre_election: true
I20260619 02:44:25.988832 10066 raft_consensus.cc:2454] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:25.988868 10338 raft_consensus.cc:2454] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 0.
I20260619 02:44:25.989219 10133 leader_election.cc:302] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c4ecf34f0c49d78820302bc2d95b17, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:25.989488 10473 raft_consensus.cc:2781] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:25.989655 10473 raft_consensus.cc:482] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:25.989770 10473 raft_consensus.cc:3037] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.992175 10473 raft_consensus.cc:504] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:25.992753 10473 leader_election.cc:288] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Requested vote from peers 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407), 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235)
I20260619 02:44:25.993137 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea92544abe4dac92796625580c1f4e" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c4ecf34f0c49d78820302bc2d95b17"
I20260619 02:44:25.993255 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea92544abe4dac92796625580c1f4e" candidate_uuid: "a0f37f6871b645a28dc31c4f266817c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01615a0d99bd4febb644911cedf39f3d"
I20260619 02:44:25.993366 10066 raft_consensus.cc:3037] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.993445 10338 raft_consensus.cc:3037] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:25.996250 10066 raft_consensus.cc:2454] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:25.996255 10338 raft_consensus.cc:2454] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0f37f6871b645a28dc31c4f266817c9 in term 1.
I20260619 02:44:25.996665 10133 leader_election.cc:302] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c4ecf34f0c49d78820302bc2d95b17, a0f37f6871b645a28dc31c4f266817c9; no voters:
I20260619 02:44:25.996934 10473 raft_consensus.cc:2781] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:25.997087 10473 raft_consensus.cc:686] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Becoming Leader. State: Replica: a0f37f6871b645a28dc31c4f266817c9, State: Running, Role: LEADER
I20260619 02:44:25.997347 10473 consensus_queue.cc:237] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } }
I20260619 02:44:26.000038 9902 catalog_manager.cc:5495] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "a0f37f6871b645a28dc31c4f266817c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0f37f6871b645a28dc31c4f266817c9" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 42825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:26.029129 10338 raft_consensus.cc:1261] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:26.029137 10066 raft_consensus.cc:1261] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Refusing update from remote peer a0f37f6871b645a28dc31c4f266817c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:26.029657 10473 consensus_queue.cc:1035] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01615a0d99bd4febb644911cedf39f3d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:26.029981 10469 consensus_queue.cc:1035] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c4ecf34f0c49d78820302bc2d95b17" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:44:26.047837 10471 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:44:27.624048 9902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:27.624269 9902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:27.627866 9902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f0169df1b147ee91594f6191c3a3a8: Sending DeleteTablet for 3 replicas of tablet d0ea92544abe4dac92796625580c1f4e
I20260619 02:44:27.628583 10180 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ea92544abe4dac92796625580c1f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:27 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260619 02:44:27.628575 10041 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ea92544abe4dac92796625580c1f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:27 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:39494
I20260619 02:44:27.628702 10319 tablet_service.cc:1514] Processing DeleteTablet for tablet d0ea92544abe4dac92796625580c1f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:27 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35814
I20260619 02:44:27.629292 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9975
I20260619 02:44:27.641335 10545 tablet_replica.cc:330] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: stopping tablet replica
I20260619 02:44:27.641673 10545 raft_consensus.cc:2229] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:27.642231 10545 raft_consensus.cc:2258] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:27.643669 10545 ts_tablet_manager.cc:1905] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:27.648874 10544 tablet_replica.cc:330] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: stopping tablet replica
I20260619 02:44:27.649140 10544 raft_consensus.cc:2229] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:27.649422 10544 raft_consensus.cc:2258] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:27.652608 10545 ts_tablet_manager.cc:1918] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:27.652776 10545 log.cc:1196] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/d0ea92544abe4dac92796625580c1f4e
I20260619 02:44:27.653228 10545 ts_tablet_manager.cc:1939] T d0ea92544abe4dac92796625580c1f4e P a0f37f6871b645a28dc31c4f266817c9: Deleting consensus metadata
I20260619 02:44:27.654798 9890 catalog_manager.cc:4841] TS a0f37f6871b645a28dc31c4f266817c9 (127.7.246.66:42825): tablet d0ea92544abe4dac92796625580c1f4e (table test-table [id=04b485702e11435b9399e639a4b0235a]) successfully deleted
I20260619 02:44:27.655217 10546 tablet_replica.cc:330] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: stopping tablet replica
I20260619 02:44:27.655445 10546 raft_consensus.cc:2229] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:27.655714 10546 raft_consensus.cc:2258] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:27.656574 10544 ts_tablet_manager.cc:1905] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:27.656626 10546 ts_tablet_manager.cc:1905] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:27.662665 10546 ts_tablet_manager.cc:1918] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:27.662822 10546 log.cc:1196] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/d0ea92544abe4dac92796625580c1f4e
I20260619 02:44:27.662883 10544 ts_tablet_manager.cc:1918] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:44:27.663057 10544 log.cc:1196] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/d0ea92544abe4dac92796625580c1f4e
I20260619 02:44:27.663302 10546 ts_tablet_manager.cc:1939] T d0ea92544abe4dac92796625580c1f4e P 01615a0d99bd4febb644911cedf39f3d: Deleting consensus metadata
I20260619 02:44:27.663542 10544 ts_tablet_manager.cc:1939] T d0ea92544abe4dac92796625580c1f4e P 80c4ecf34f0c49d78820302bc2d95b17: Deleting consensus metadata
I20260619 02:44:27.664322 9888 catalog_manager.cc:4841] TS 01615a0d99bd4febb644911cedf39f3d (127.7.246.67:34235): tablet d0ea92544abe4dac92796625580c1f4e (table test-table [id=04b485702e11435b9399e639a4b0235a]) successfully deleted
I20260619 02:44:27.664641 9890 catalog_manager.cc:4841] TS 80c4ecf34f0c49d78820302bc2d95b17 (127.7.246.65:33407): tablet d0ea92544abe4dac92796625580c1f4e (table test-table [id=04b485702e11435b9399e639a4b0235a]) successfully deleted
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:27.671260 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:27.631318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.7.246.65:33407)
0619 02:44:27.654883 (+ 23565us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:27.654890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:27.654975 (+ 85us) server_negotiation.cc:407] Connection header received
0619 02:44:27.655042 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:27.655046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:27.655184 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:27.655351 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:27.656959 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.658171 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.660096 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.660754 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.660885 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:27.665890 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:27.665905 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:27.665921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:27.665953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:27.668452 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.669132 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.669140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.669142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.669279 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.669720 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.669725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.669730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.670085 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:27.670393 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:27.670670 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:44:27.670893 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23407,"thread_start_us":23277,"threads_started":1}
W20260619 02:44:27.672282 10106 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39506
W20260619 02:44:27.672843 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9975: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:27.673086 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9975
I20260619 02:44:27.692576 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10115
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:27.708274 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:27.693343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.7.246.66:42825)
0619 02:44:27.693947 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:27.693952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:27.693967 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:44:27.694019 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:27.694023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:27.694088 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:27.694176 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:27.695333 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.696094 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.697268 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.697783 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.697877 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:27.701813 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:27.701827 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:27.701843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:27.701882 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:27.705167 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.705970 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.705985 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.705988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.706187 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.706557 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.706562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.706566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.706957 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:27.707265 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:27.707634 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 02:44:27.707853 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":369,"threads_started":1}
W20260619 02:44:27.709066 10245 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44002
W20260619 02:44:27.709771 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:27.709926 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10115
I20260619 02:44:27.735498 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10254
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[9859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:27.750427 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:27.736217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.7.246.67:34235)
0619 02:44:27.736691 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:27.736695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:27.736710 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:44:27.736761 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:27.736764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:27.736818 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:27.736910 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:27.738237 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.738968 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.740133 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:27.740504 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:27.740605 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:27.744772 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:27.744786 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:27.744803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:27.744837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:27.747607 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.748229 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.748237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.748239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.748351 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:27.748752 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:27.748758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:27.748767 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:27.749123 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:27.749373 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:27.749910 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 02:44:27.750141 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":220,"threads_started":1}
W20260619 02:44:27.751093 10384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35828
W20260619 02:44:27.751659 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:27.751804 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10254
I20260619 02:44:27.773890 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 9872
I20260619 02:44:27.775058 9944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:27.889990 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 9872
2026-06-19T02:44:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7988 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[10554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[10554](info): set up 2 sockets
Jun 19 02:44:27 dist-test-slave-9tkz krb5kdc[10554](info): commencing operation
krb5kdc: starting...
W20260619 02:44:28.987187 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:29 dist-test-slave-9tkz krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:29.186533 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:35743
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:35743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:29.500998 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:29.501296 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:29.501473 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:29.508716 10567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:29.508833 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:29.508880 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:29.508958 10567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:29.509021 10567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:29.519024 10567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:35743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:29.520347 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:29.521791 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:29.534220 10572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:29.534137 10573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:29.534160 10575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:29.535128 10567 server_base.cc:1034] running on GCE node
I20260619 02:44:29.535909 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:29.537348 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:29.538616 10567 hybrid_clock.cc:648] HybridClock initialized: now 1781837069538573 us; error 49 us; skew 500 ppm
I20260619 02:44:29.539034 10567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:29 dist-test-slave-9tkz krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837069, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:29.542349 10567 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:29.543948 10567 webserver.cc:460] Webserver started at http://127.7.246.126:45247/ using document root <none> and password file <none>
I20260619 02:44:29.544662 10567 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:29.544809 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:29.545135 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:29.548219 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "4dce91c08c8945918fd471f5eab4ee38"
format_stamp: "Formatted at 2026-06-19 02:44:29 on dist-test-slave-9tkz"
server_key: "cc2a24ff52b69b4b2493fc51fac2dbc3"
server_key_iv: "a842d1a5bee24114d222cc9318bc0153"
server_key_version: "encryptionkey@0"
I20260619 02:44:29.549041 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "4dce91c08c8945918fd471f5eab4ee38"
format_stamp: "Formatted at 2026-06-19 02:44:29 on dist-test-slave-9tkz"
server_key: "cc2a24ff52b69b4b2493fc51fac2dbc3"
server_key_iv: "a842d1a5bee24114d222cc9318bc0153"
server_key_version: "encryptionkey@0"
I20260619 02:44:29.554263 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:44:29.557765 10582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:29.559192 10567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:44:29.559389 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "4dce91c08c8945918fd471f5eab4ee38"
format_stamp: "Formatted at 2026-06-19 02:44:29 on dist-test-slave-9tkz"
server_key: "cc2a24ff52b69b4b2493fc51fac2dbc3"
server_key_iv: "a842d1a5bee24114d222cc9318bc0153"
server_key_version: "encryptionkey@0"
I20260619 02:44:29.559614 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:29.573076 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:29.578073 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:29.578394 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:29.596848 10567 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:35743
I20260619 02:44:29.596854 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:35743 every 8 connection(s)
I20260619 02:44:29.598366 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:29.602164 10634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:29.603897 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10567
I20260619 02:44:29.604125 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:29.604909 8153 external_mini_cluster.cc:1428] Setting key e6000ed5789cb1610eb9d67bd0e8f1e9
I20260619 02:44:29.614836 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38: Bootstrap starting.
I20260619 02:44:29.618575 10634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:29.619832 10634 log.cc:824] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:44:29 dist-test-slave-9tkz krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:29.622753 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38: No bootstrap required, opened a new log
I20260619 02:44:29.628837 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:29.607098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.7.246.126:35743)
0619 02:44:29.607984 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:29.607997 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:29.608045 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:44:29.609902 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:29.609926 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:29.610346 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:29.610688 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:29.612480 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:29.613910 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:29.615693 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:29.616417 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:29.616553 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:29.621461 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:29.621469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:29.621483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:29.621517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:29.624706 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:29.625444 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:29.625459 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:29.625465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:29.625702 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:29.626119 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:29.626128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:29.626132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:29.626666 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:29.627001 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:29.627419 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 02:44:29.627849 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":418,"threads_started":1}
I20260619 02:44:29.630487 10634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } }
I20260619 02:44:29.630842 10634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:29.630970 10634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dce91c08c8945918fd471f5eab4ee38, State: Initialized, Role: FOLLOWER
I20260619 02:44:29.631541 10634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } }
I20260619 02:44:29.631779 10634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:29.631925 10634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:29.632094 10634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:29.634276 10634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } }
I20260619 02:44:29.634743 10634 leader_election.cc:302] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dce91c08c8945918fd471f5eab4ee38; no voters:
I20260619 02:44:29.635336 10634 leader_election.cc:288] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:29.635466 10639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:29.636166 10639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [term 1 LEADER]: Becoming Leader. State: Replica: 4dce91c08c8945918fd471f5eab4ee38, State: Running, Role: LEADER
I20260619 02:44:29.636763 10639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } }
I20260619 02:44:29.637445 10634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:29.639772 10641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dce91c08c8945918fd471f5eab4ee38. Latest consensus state: current_term: 1 leader_uuid: "4dce91c08c8945918fd471f5eab4ee38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } } }
I20260619 02:44:29.639747 10640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dce91c08c8945918fd471f5eab4ee38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dce91c08c8945918fd471f5eab4ee38" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35743 } } }
I20260619 02:44:29.640214 10641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:29.640214 10640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:29.641047 10644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:29.648813 10644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:29.657195 10644 catalog_manager.cc:1347] Generated new cluster ID: dccc607e87e04c17b76ceba737031fd0
I20260619 02:44:29.657346 10644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:29.672114 10644 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:29.673475 10644 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:29.684180 10644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dce91c08c8945918fd471f5eab4ee38: Generated new TSK 0
I20260619 02:44:29.685268 10644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:29.727238 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:30.054654 10660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:30.054919 10660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:30.055008 10660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:30.062358 10660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:30.062485 10660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:30.062711 10660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:30.072669 10660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:30.074122 10660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:30.075451 10660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:30.086964 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:30.087769 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:30.088495 10668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:30.089409 10660 server_base.cc:1034] running on GCE node
I20260619 02:44:30.090144 10660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:30.091046 10660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:30.092279 10660 hybrid_clock.cc:648] HybridClock initialized: now 1781837070092235 us; error 53 us; skew 500 ppm
I20260619 02:44:30.092599 10660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:30 dist-test-slave-9tkz krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837070, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:30.095759 10660 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:30.097189 10660 webserver.cc:460] Webserver started at http://127.7.246.65:46221/ using document root <none> and password file <none>
I20260619 02:44:30.097910 10660 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:30.098009 10660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:30.098271 10660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:30.101006 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "c910625f2de2479e9ce045943c45727b"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "bf4f3e44db70a76a3d74e275ab36b659"
server_key_iv: "9901907e2a278c388a58dab95086f14b"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.101805 10660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "c910625f2de2479e9ce045943c45727b"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "bf4f3e44db70a76a3d74e275ab36b659"
server_key_iv: "9901907e2a278c388a58dab95086f14b"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.106865 10660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:30.110368 10675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.111676 10660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:30.111853 10660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "c910625f2de2479e9ce045943c45727b"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "bf4f3e44db70a76a3d74e275ab36b659"
server_key_iv: "9901907e2a278c388a58dab95086f14b"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.112043 10660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:30.129805 10660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:30.133536 10660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:30.133898 10660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:30.134824 10660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:30.136222 10660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:30.136291 10660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.136474 10660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:30.136533 10660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.169320 10660 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36415
I20260619 02:44:30.169378 10787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36415 every 8 connection(s)
I20260619 02:44:30.170624 10660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:30.175801 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10660
I20260619 02:44:30.176002 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:30.176756 8153 external_mini_cluster.cc:1428] Setting key 9565146ef15a8d40175ec85f811c9c73
Jun 19 02:44:30 dist-test-slave-9tkz krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837070, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:30.193822 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:30.173086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:43705 (local address 127.7.246.126:35743)
0619 02:44:30.173526 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:30.173531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:30.174594 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 02:44:30.176953 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:30.176958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:30.177022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:30.177119 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:30.178948 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.179761 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.182292 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.182745 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.182849 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:30.186583 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:30.186593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:30.186601 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:30.186657 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:30.190337 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:30.191053 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:30.191062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:30.191069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:30.191232 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:30.191642 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:30.191650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:30.191656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:30.192086 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:30.192271 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:30.192962 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 02:44:30.193232 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":244,"threads_started":1}
I20260619 02:44:30.194379 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:30.173786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:35743 (local address 127.7.246.65:43705)
0619 02:44:30.174439 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:30.174483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:30.176659 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:30.177401 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:30.177409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:30.178040 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:30.178758 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:30.178784 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.179928 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.179937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:30.182137 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:30.182154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.182964 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.182972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:30.183310 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:30.184132 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:30.184158 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:30.186301 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:30.190475 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:30.190483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:30.190496 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:30.190901 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:30.191343 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:30.191346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:30.191348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:30.191515 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:30.192247 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:30.192257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:30.192580 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 02:44:30.192799 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 02:44:30.193172 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":250,"threads_started":1}
I20260619 02:44:30.195941 10788 heartbeater.cc:347] Connected to a master server at 127.7.246.126:35743
I20260619 02:44:30.196311 10788 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:30.197324 10788 heartbeater.cc:511] Master 127.7.246.126:35743 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:44:30.199781 10599 ts_manager.cc:194] Registered new tserver with Master: c910625f2de2479e9ce045943c45727b (127.7.246.65:36415)
I20260619 02:44:30.201969 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:43705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:30.223068 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:30.540967 10794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:30.541323 10794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:30.541477 10794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:30.548849 10794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:30.549018 10794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:30.549144 10794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:30.558923 10794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:30.560228 10794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:30.561513 10794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:30.575304 10799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:30.575325 10802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:30.575251 10800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:30.576094 10794 server_base.cc:1034] running on GCE node
I20260619 02:44:30.576719 10794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:30.577752 10794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:30.578954 10794 hybrid_clock.cc:648] HybridClock initialized: now 1781837070578911 us; error 38 us; skew 500 ppm
I20260619 02:44:30.579278 10794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:30 dist-test-slave-9tkz krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837070, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:30.583380 10794 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:30.587495 10794 webserver.cc:460] Webserver started at http://127.7.246.66:44461/ using document root <none> and password file <none>
I20260619 02:44:30.588569 10794 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:30.588703 10794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:30.589100 10794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:30.593253 10794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "5256f722b9024a3bba1c31140d9a1590"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "54a420a9d4f0c653ca148234dbf46329"
server_key_iv: "4fb20f6d27adbeec805319f0cd1da0c3"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.594406 10794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "5256f722b9024a3bba1c31140d9a1590"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "54a420a9d4f0c653ca148234dbf46329"
server_key_iv: "4fb20f6d27adbeec805319f0cd1da0c3"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.601915 10794 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 02:44:30.606983 10809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.608776 10794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260619 02:44:30.608968 10794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "5256f722b9024a3bba1c31140d9a1590"
format_stamp: "Formatted at 2026-06-19 02:44:30 on dist-test-slave-9tkz"
server_key: "54a420a9d4f0c653ca148234dbf46329"
server_key_iv: "4fb20f6d27adbeec805319f0cd1da0c3"
server_key_version: "encryptionkey@0"
I20260619 02:44:30.609246 10794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:30.627918 10794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:30.632427 10794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:30.632717 10794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:30.633829 10794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:30.635134 10794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:30.635257 10794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.635406 10794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:30.635476 10794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:30.668994 10794 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:36063
I20260619 02:44:30.669041 10921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:36063 every 8 connection(s)
I20260619 02:44:30.670372 10794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:30.670884 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10794
I20260619 02:44:30.671093 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:30.671666 8153 external_mini_cluster.cc:1428] Setting key 7e8e0a83fedaec79e03ea81ef1de4903
Jun 19 02:44:30 dist-test-slave-9tkz krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837070, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:44:30.693341 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:30.673230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:41421 (local address 127.7.246.126:35743)
0619 02:44:30.673434 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:30.673440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:30.674936 (+ 1496us) server_negotiation.cc:407] Connection header received
0619 02:44:30.677831 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:30.677836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:30.677900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:30.677985 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:30.680142 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.680901 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.682546 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.682932 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.683067 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:30.686775 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:30.686785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:30.686791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:30.686830 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:30.690329 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:30.691065 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:30.691074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:30.691080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:30.691255 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:30.691660 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:30.691663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:30.691665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:30.691946 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:30.692091 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:30.692790 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 02:44:30.693005 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:44:30.694459 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:30.673869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:35743 (local address 127.7.246.66:41421)
0619 02:44:30.674749 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:30.674804 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:30.677498 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:30.678294 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:30.678307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:30.679113 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:30.679939 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:30.679968 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.681083 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.681088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:30.682389 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:30.682400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:30.683170 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:30.683176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:30.683402 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:30.684125 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:30.684147 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:30.686508 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:30.690473 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:30.690480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:30.690493 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:30.690921 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:30.691378 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:30.691381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:30.691383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:30.691554 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:30.692086 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:30.692094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:30.692438 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 02:44:30.692710 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 02:44:30.693327 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":316,"threads_started":1}
I20260619 02:44:30.696174 10922 heartbeater.cc:347] Connected to a master server at 127.7.246.126:35743
I20260619 02:44:30.696483 10922 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:30.697227 10922 heartbeater.cc:511] Master 127.7.246.126:35743 requested a full tablet report, sending...
I20260619 02:44:30.698853 10599 ts_manager.cc:194] Registered new tserver with Master: 5256f722b9024a3bba1c31140d9a1590 (127.7.246.66:36063)
I20260619 02:44:30.700155 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:41421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:30.720074 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:31.040225 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:31.040508 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:31.040613 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:31.047822 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:31.047927 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:31.048036 10927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:31.057698 10927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:35743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:31.058965 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:31.060266 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:31.073520 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:31.073593 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:31.073643 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:31.074374 10927 server_base.cc:1034] running on GCE node
I20260619 02:44:31.075004 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:31.075932 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:31.077157 10927 hybrid_clock.cc:648] HybridClock initialized: now 1781837071077107 us; error 60 us; skew 500 ppm
I20260619 02:44:31.077469 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:31 dist-test-slave-9tkz krb5kdc[10554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837071, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:31.080576 10927 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:31.082100 10927 webserver.cc:460] Webserver started at http://127.7.246.67:34279/ using document root <none> and password file <none>
I20260619 02:44:31.082784 10927 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:31.082927 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:31.083235 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:31.086339 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "4db4ee4141d44bbcbc5ac9356046aec9"
format_stamp: "Formatted at 2026-06-19 02:44:31 on dist-test-slave-9tkz"
server_key: "d5a7e7b6703bdfa183488021051cfc2e"
server_key_iv: "bc367297d5c217a7949ca602e943ad9d"
server_key_version: "encryptionkey@0"
I20260619 02:44:31.087158 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "4db4ee4141d44bbcbc5ac9356046aec9"
format_stamp: "Formatted at 2026-06-19 02:44:31 on dist-test-slave-9tkz"
server_key: "d5a7e7b6703bdfa183488021051cfc2e"
server_key_iv: "bc367297d5c217a7949ca602e943ad9d"
server_key_version: "encryptionkey@0"
I20260619 02:44:31.092260 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:31.095739 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:31.097124 10927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:31.097311 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "4db4ee4141d44bbcbc5ac9356046aec9"
format_stamp: "Formatted at 2026-06-19 02:44:31 on dist-test-slave-9tkz"
server_key: "d5a7e7b6703bdfa183488021051cfc2e"
server_key_iv: "bc367297d5c217a7949ca602e943ad9d"
server_key_version: "encryptionkey@0"
I20260619 02:44:31.097515 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:31.122575 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:31.127110 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:31.127409 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:31.128352 10927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:31.129690 10927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:31.129786 10927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:31.129927 10927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:31.129995 10927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:31.162371 10927 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:41163
I20260619 02:44:31.162416 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:41163 every 8 connection(s)
I20260619 02:44:31.163810 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:31.167938 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 10927
I20260619 02:44:31.168128 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:31.168591 8153 external_mini_cluster.cc:1428] Setting key ff8dcd9c5a11f58ba962aa0b2f36d604
Jun 19 02:44:31 dist-test-slave-9tkz krb5kdc[10554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837071, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:31.184459 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:31.166442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:41973 (local address 127.7.246.126:35743)
0619 02:44:31.166605 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:31.166609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:31.167789 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 02:44:31.169949 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:31.169954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:31.170059 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:31.170157 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:31.172320 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:31.173071 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:31.174661 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:31.175084 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:31.175230 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:31.178627 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:31.178632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:31.178636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:31.178669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:31.181263 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:31.181972 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:31.181976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:31.181978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:31.182117 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:31.182633 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:31.182637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:31.182640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:31.182934 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:31.183041 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:31.183906 (+ 865us) server_negotiation.cc:299] Negotiation successful
0619 02:44:31.184111 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:44:31.185516 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:31.166917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:35743 (local address 127.7.246.67:41973)
0619 02:44:31.167639 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:31.167674 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:31.169643 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:31.170581 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:31.170593 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:31.171303 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:31.172158 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:31.172187 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:31.173235 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:31.173241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:31.174519 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:31.174531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:31.175302 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:31.175311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:31.175555 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:31.176188 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:31.176212 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:31.178428 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:31.181381 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:31.181387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:31.181401 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:31.181865 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:31.182223 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:31.182229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:31.182232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:31.182465 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:31.183060 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:31.183068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:31.183483 (+ 415us) client_negotiation.cc:769] Sending connection context
0619 02:44:31.183817 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 02:44:31.184349 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260619 02:44:31.186829 11055 heartbeater.cc:347] Connected to a master server at 127.7.246.126:35743
I20260619 02:44:31.187095 11055 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:31.187736 11055 heartbeater.cc:511] Master 127.7.246.126:35743 requested a full tablet report, sending...
I20260619 02:44:31.189090 10599 ts_manager.cc:194] Registered new tserver with Master: 4db4ee4141d44bbcbc5ac9356046aec9 (127.7.246.67:41163)
I20260619 02:44:31.190002 10599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:41973
I20260619 02:44:31.196792 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:31.205149 10788 heartbeater.cc:503] Master 127.7.246.126:35743 was elected leader, sending a full tablet report...
W20260619 02:44:31.227499 10790 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:31.225232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.7.246.126:35743)
0619 02:44:31.225399 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:31.225403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:31.225940 (+ 537us) server_negotiation.cc:407] Connection header received
0619 02:44:31.226087 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:31.226091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:31.226150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:31.226240 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:31.227305 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41100: BlockingRecv error: recv got EOF from 127.0.0.1:41100 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:44:31.230975 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10660
W20260619 02:44:31.233830 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10660: Not authorized: Client connection negotiation failed: client connection to 127.7.246.65:36415: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:44:31.234004 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10660
I20260619 02:44:31.246577 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10794
W20260619 02:44:31.249059 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10794: Not authorized: Client connection negotiation failed: client connection to 127.7.246.66:36063: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:44:31.249204 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10794
I20260619 02:44:31.264379 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10927
W20260619 02:44:31.267371 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10927: Not authorized: Client connection negotiation failed: client connection to 127.7.246.67:41163: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:44:31.267540 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10927
I20260619 02:44:31.282641 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 10567
I20260619 02:44:31.283991 10629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:31.384265 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 10567
2026-06-19T02:44:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3491 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:31 dist-test-slave-9tkz krb5kdc[11072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:31 dist-test-slave-9tkz krb5kdc[11072](info): set up 2 sockets
Jun 19 02:44:31 dist-test-slave-9tkz krb5kdc[11072](info): commencing operation
krb5kdc: starting...
W20260619 02:44:32.465957 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:32 dist-test-slave-9tkz krb5kdc[11072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:32.656908 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:37693
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:37693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:32.970784 11085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:32.971066 11085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:32.971241 11085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:32.978435 11085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:32.978564 11085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:32.978672 11085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:32.978766 11085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:32.978852 11085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:32.988878 11085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:37693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:32.990265 11085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:32.991641 11085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:33.003897 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:33.003902 11091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:33.003938 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:33.004782 11085 server_base.cc:1034] running on GCE node
I20260619 02:44:33.005620 11085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:33.006991 11085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:33.008215 11085 hybrid_clock.cc:648] HybridClock initialized: now 1781837073008180 us; error 57 us; skew 500 ppm
I20260619 02:44:33.008549 11085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:33 dist-test-slave-9tkz krb5kdc[11072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837073, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:33.011814 11085 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:33.013238 11085 webserver.cc:460] Webserver started at http://127.7.246.126:43975/ using document root <none> and password file <none>
I20260619 02:44:33.013983 11085 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:33.014117 11085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:33.014417 11085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:33.017302 11085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7"
format_stamp: "Formatted at 2026-06-19 02:44:33 on dist-test-slave-9tkz"
server_key: "6090f4af76152445239de1081ba17426"
server_key_iv: "3c85a2c6bf616bebadf97cd2b73db73c"
server_key_version: "encryptionkey@0"
I20260619 02:44:33.018118 11085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7"
format_stamp: "Formatted at 2026-06-19 02:44:33 on dist-test-slave-9tkz"
server_key: "6090f4af76152445239de1081ba17426"
server_key_iv: "3c85a2c6bf616bebadf97cd2b73db73c"
server_key_version: "encryptionkey@0"
I20260619 02:44:33.022993 11085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:44:33.026364 11100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:33.027679 11085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:33.027848 11085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7"
format_stamp: "Formatted at 2026-06-19 02:44:33 on dist-test-slave-9tkz"
server_key: "6090f4af76152445239de1081ba17426"
server_key_iv: "3c85a2c6bf616bebadf97cd2b73db73c"
server_key_version: "encryptionkey@0"
I20260619 02:44:33.028040 11085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:33.041309 11085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:33.045073 11085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:33.045353 11085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:33.062717 11085 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:37693
I20260619 02:44:33.062747 11151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:37693 every 8 connection(s)
I20260619 02:44:33.064162 11085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:33.064594 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11085
I20260619 02:44:33.064819 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:33.065744 8153 external_mini_cluster.cc:1428] Setting key 4abade855c3f0e6f09b7cb22318b5e0c
I20260619 02:44:33.069056 11152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:44:33 dist-test-slave-9tkz krb5kdc[11072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:33.082015 11152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7: Bootstrap starting.
I20260619 02:44:33.085191 11152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:33.086457 11152 log.cc:824] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:33.088474 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:33.067794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.7.246.126:37693)
0619 02:44:33.068611 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:33.068622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:33.068661 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:33.070312 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:33.070334 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:33.070704 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:33.071010 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:33.072794 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:33.074195 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:33.075885 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:33.076479 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:33.076643 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:33.081090 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:33.081097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:33.081110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:33.081148 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:33.084457 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:33.085102 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:33.085111 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:33.085121 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:33.085278 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:33.085763 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:33.085766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:33.085768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:33.086265 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:33.086457 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:33.086916 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 02:44:33.087390 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":392,"threads_started":1}
I20260619 02:44:33.089638 11152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7: No bootstrap required, opened a new log
I20260619 02:44:33.092262 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:44:33.097067 11152 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } }
I20260619 02:44:33.097450 11152 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:33.097638 11152 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f1764e9ae54b279f7dc2dc068cb2a7, State: Initialized, Role: FOLLOWER
I20260619 02:44:33.098269 11152 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } }
I20260619 02:44:33.098521 11152 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:33.098672 11152 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:33.098850 11152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:33.102362 11152 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } }
I20260619 02:44:33.102954 11152 leader_election.cc:302] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0f1764e9ae54b279f7dc2dc068cb2a7; no voters:
I20260619 02:44:33.103617 11152 leader_election.cc:288] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:33.103726 11158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:33.104985 11158 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [term 1 LEADER]: Becoming Leader. State: Replica: c0f1764e9ae54b279f7dc2dc068cb2a7, State: Running, Role: LEADER
I20260619 02:44:33.105578 11158 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } }
I20260619 02:44:33.105870 11152 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:33.108273 11159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } } }
I20260619 02:44:33.108817 11159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:33.108644 11160 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0f1764e9ae54b279f7dc2dc068cb2a7. Latest consensus state: current_term: 1 leader_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0f1764e9ae54b279f7dc2dc068cb2a7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 37693 } } }
I20260619 02:44:33.108976 11160 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:33.111871 11166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:33.121857 11166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:33.135177 11166 catalog_manager.cc:1347] Generated new cluster ID: e1637ab554d84c268b65795867492e95
I20260619 02:44:33.135347 11166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:33.165746 11166 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:33.168126 11166 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:33.176702 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:33.157667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.7.246.126:37693)
0619 02:44:33.157849 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:33.157855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:33.167392 (+ 9537us) server_negotiation.cc:407] Connection header received
0619 02:44:33.167609 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:33.167615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:33.168138 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:33.168276 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:33.169654 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:33.170850 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:33.172813 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:33.173520 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:33.173671 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:33.174908 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:33.174914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:33.174919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:33.175820 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:33.175878 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:33.175905 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:33.176011 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:33.176188 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:44:33.176285 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:44:33.189062 11166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0f1764e9ae54b279f7dc2dc068cb2a7: Generated new TSK 0
I20260619 02:44:33.190061 11166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:44:33.199349 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11085
I20260619 02:44:33.200461 11147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:33.288452 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11085
2026-06-19T02:44:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1895 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:33 dist-test-slave-9tkz krb5kdc[11186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:33 dist-test-slave-9tkz krb5kdc[11186](info): set up 2 sockets
Jun 19 02:44:33 dist-test-slave-9tkz krb5kdc[11186](info): commencing operation
krb5kdc: starting...
W20260619 02:44:34.391835 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 02:44:34.634586 8153 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.243s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 02:44:34.852653 8153 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.218s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 02:44:35.152405 8153 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.300s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:35 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:35.320681 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41591
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:35.646695 11199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:35.646994 11199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:35.647081 11199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:35.654641 11199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:35.654762 11199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:35.654810 11199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:35.654858 11199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:35.654902 11199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:35.664837 11199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:35.666188 11199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:35.667408 11199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:35.679688 11205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:35.679802 11204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:35.679781 11207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:35.680509 11199 server_base.cc:1034] running on GCE node
I20260619 02:44:35.681365 11199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:35.682884 11199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:35.684108 11199 hybrid_clock.cc:648] HybridClock initialized: now 1781837075684061 us; error 47 us; skew 500 ppm
I20260619 02:44:35.684458 11199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:35 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837075, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:35.693071 11199 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:35.694731 11199 webserver.cc:460] Webserver started at http://127.7.246.126:46333/ using document root <none> and password file <none>
I20260619 02:44:35.695490 11199 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:35.695643 11199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:35.695986 11199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:35.699083 11199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "772436df1fff42cdb11a9b223e81d82f"
format_stamp: "Formatted at 2026-06-19 02:44:35 on dist-test-slave-9tkz"
server_key: "48afee93b9477def6cfac9e61d27bad6"
server_key_iv: "bf8aae27397e786b64874bb5f5806ec0"
server_key_version: "encryptionkey@0"
I20260619 02:44:35.699857 11199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "772436df1fff42cdb11a9b223e81d82f"
format_stamp: "Formatted at 2026-06-19 02:44:35 on dist-test-slave-9tkz"
server_key: "48afee93b9477def6cfac9e61d27bad6"
server_key_iv: "bf8aae27397e786b64874bb5f5806ec0"
server_key_version: "encryptionkey@0"
I20260619 02:44:35.705462 11199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 02:44:35.709045 11214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:35.710448 11199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:44:35.710660 11199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "772436df1fff42cdb11a9b223e81d82f"
format_stamp: "Formatted at 2026-06-19 02:44:35 on dist-test-slave-9tkz"
server_key: "48afee93b9477def6cfac9e61d27bad6"
server_key_iv: "bf8aae27397e786b64874bb5f5806ec0"
server_key_version: "encryptionkey@0"
I20260619 02:44:35.710865 11199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:35.726809 11199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:35.730903 11199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:35.731206 11199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:35.750105 11199 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41591
I20260619 02:44:35.750128 11265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41591 every 8 connection(s)
I20260619 02:44:35.751554 11199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:35.755556 11266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:35.759042 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11199
I20260619 02:44:35.759284 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:35.760076 8153 external_mini_cluster.cc:1428] Setting key 6285c4b9936d57c546d0e3cc370d90fc
I20260619 02:44:35.768237 11266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f: Bootstrap starting.
I20260619 02:44:35.771428 11266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:35.772475 11266 log.cc:824] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:44:35 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:35.775753 11266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f: No bootstrap required, opened a new log
I20260619 02:44:35.783710 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:35.762327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.7.246.126:41591)
0619 02:44:35.763153 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:35.763165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:35.763211 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:44:35.764864 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:35.764889 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:35.765336 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:35.765815 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:35.767249 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:35.768585 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:35.769875 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:35.770587 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:35.770780 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:35.774857 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:35.774867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:35.774885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:35.774925 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:35.778912 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:35.779609 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:35.779618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:35.779627 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:35.779829 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:35.780192 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:35.780197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:35.780201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:35.780938 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:35.781379 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:35.781769 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 02:44:35.782237 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260619 02:44:35.786752 11266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } }
I20260619 02:44:35.787158 11266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:35.787236 11266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772436df1fff42cdb11a9b223e81d82f, State: Initialized, Role: FOLLOWER
I20260619 02:44:35.787757 11266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } }
I20260619 02:44:35.787937 11266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:35.788060 11266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:35.788229 11266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:35.790676 11266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } }
I20260619 02:44:35.791172 11266 leader_election.cc:302] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 772436df1fff42cdb11a9b223e81d82f; no voters:
I20260619 02:44:35.791975 11266 leader_election.cc:288] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:35.792030 11271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:35.793360 11271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [term 1 LEADER]: Becoming Leader. State: Replica: 772436df1fff42cdb11a9b223e81d82f, State: Running, Role: LEADER
I20260619 02:44:35.793921 11271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } }
I20260619 02:44:35.794410 11266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:35.797115 11272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "772436df1fff42cdb11a9b223e81d82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } } }
I20260619 02:44:35.797223 11273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 772436df1fff42cdb11a9b223e81d82f. Latest consensus state: current_term: 1 leader_uuid: "772436df1fff42cdb11a9b223e81d82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772436df1fff42cdb11a9b223e81d82f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41591 } } }
I20260619 02:44:35.797632 11273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:35.797644 11272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:35.799352 11279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:35.805048 11279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:35.814324 11279 catalog_manager.cc:1347] Generated new cluster ID: 340adc208b89479abfc3fd524af1c747
I20260619 02:44:35.814465 11279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:35.846053 11279 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:35.847388 11279 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:35.860780 11279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 772436df1fff42cdb11a9b223e81d82f: Generated new TSK 0
I20260619 02:44:35.861722 11279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:35.917052 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:36.236831 11292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:36.237104 11292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:36.237259 11292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:36.244544 11292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:36.244654 11292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:36.244841 11292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:36.254735 11292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:36.255999 11292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:36.257277 11292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:36.269966 11297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:36.269912 11300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:36.269873 11298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:36.270584 11292 server_base.cc:1034] running on GCE node
I20260619 02:44:36.271162 11292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:36.272073 11292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:36.273306 11292 hybrid_clock.cc:648] HybridClock initialized: now 1781837076273253 us; error 49 us; skew 500 ppm
I20260619 02:44:36.273669 11292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:36 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837076, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:36.276690 11292 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:36.278144 11292 webserver.cc:460] Webserver started at http://127.7.246.65:36131/ using document root <none> and password file <none>
I20260619 02:44:36.278851 11292 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:36.278992 11292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:36.279290 11292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:36.282160 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "9cc92cbfb5fc4875877c6c214344fa1b"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "51b5c94713dcf76c86a632925d12451e"
server_key_iv: "14fccf0340c86268ccd99be11a3e025c"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.282932 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "9cc92cbfb5fc4875877c6c214344fa1b"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "51b5c94713dcf76c86a632925d12451e"
server_key_iv: "14fccf0340c86268ccd99be11a3e025c"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.287957 11292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:36.291245 11307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.292519 11292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:44:36.292689 11292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "9cc92cbfb5fc4875877c6c214344fa1b"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "51b5c94713dcf76c86a632925d12451e"
server_key_iv: "14fccf0340c86268ccd99be11a3e025c"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.292882 11292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:36.307168 11292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:36.311053 11292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:36.311339 11292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:36.312251 11292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:36.313505 11292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:36.313632 11292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.313791 11292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:36.313853 11292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.347563 11292 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36429
I20260619 02:44:36.347658 11419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36429 every 8 connection(s)
I20260619 02:44:36.348819 11292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:36.355592 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11292
I20260619 02:44:36.355829 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:36.356585 8153 external_mini_cluster.cc:1428] Setting key 7b9fe36d39f6dd46ac8c18b877386f34
Jun 19 02:44:36 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837076, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:36.371203 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:36.351577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:59177 (local address 127.7.246.126:41591)
0619 02:44:36.351996 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:36.352000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:36.353113 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 02:44:36.354943 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:36.354947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:36.355010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:36.355129 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:36.357239 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.358345 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.360844 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.361253 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.361348 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:36.365399 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:36.365408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:36.365413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:36.365453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:36.368024 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:36.368937 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:36.368941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:36.368943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:36.369044 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:36.369523 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:36.369526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:36.369528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:36.369827 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:36.369943 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:36.370802 (+ 859us) server_negotiation.cc:299] Negotiation successful
0619 02:44:36.370953 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":214,"threads_started":1}
I20260619 02:44:36.372862 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:36.352164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41591 (local address 127.7.246.65:59177)
0619 02:44:36.352940 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:36.352993 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:36.354627 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:36.355405 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:36.355416 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:36.356160 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:36.356989 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:36.357013 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.358516 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.358521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:36.360682 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:36.360699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.361471 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.361480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:36.361839 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:36.362571 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:36.362604 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:36.365168 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:36.368170 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:36.368182 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:36.368201 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:36.368831 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:36.369148 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:36.369156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:36.369162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:36.369417 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:36.369955 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:36.369966 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:36.370415 (+ 449us) client_negotiation.cc:769] Sending connection context
0619 02:44:36.370749 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 02:44:36.371342 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":332,"threads_started":1}
I20260619 02:44:36.374580 11420 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41591
I20260619 02:44:36.374928 11420 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:36.375633 11420 heartbeater.cc:511] Master 127.7.246.126:41591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:36.377651 11231 ts_manager.cc:194] Registered new tserver with Master: 9cc92cbfb5fc4875877c6c214344fa1b (127.7.246.65:36429)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:44:36.379359 11231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:59177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:36.399987 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:36.721877 11426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:36.722160 11426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:36.722254 11426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:36.731140 11426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:36.731258 11426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:36.731370 11426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:36.741235 11426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:36.742532 11426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:36.743710 11426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:36.757088 11434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:36.757055 11432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:36.757573 11431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:36.758073 11426 server_base.cc:1034] running on GCE node
I20260619 02:44:36.758705 11426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:36.759661 11426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:36.760893 11426 hybrid_clock.cc:648] HybridClock initialized: now 1781837076760833 us; error 54 us; skew 500 ppm
I20260619 02:44:36.761229 11426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:36 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837076, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:36.764539 11426 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:36.766089 11426 webserver.cc:460] Webserver started at http://127.7.246.66:39825/ using document root <none> and password file <none>
I20260619 02:44:36.766793 11426 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:36.766932 11426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:36.767225 11426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:36.770035 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "2a32a1b33a8a4aba9f2b6913e740d9a4"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "d5c7d2e1fe7655bd376bfd53c012edc6"
server_key_iv: "5fd32fdcc179d4e2f7140d3d8ca5c4f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.770815 11426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "2a32a1b33a8a4aba9f2b6913e740d9a4"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "d5c7d2e1fe7655bd376bfd53c012edc6"
server_key_iv: "5fd32fdcc179d4e2f7140d3d8ca5c4f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.775861 11426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 02:44:36.779436 11441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.780823 11426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:36.781001 11426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "2a32a1b33a8a4aba9f2b6913e740d9a4"
format_stamp: "Formatted at 2026-06-19 02:44:36 on dist-test-slave-9tkz"
server_key: "d5c7d2e1fe7655bd376bfd53c012edc6"
server_key_iv: "5fd32fdcc179d4e2f7140d3d8ca5c4f5"
server_key_version: "encryptionkey@0"
I20260619 02:44:36.781200 11426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:36.805207 11426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:36.809900 11426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:36.810196 11426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:36.811162 11426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:36.812470 11426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:36.812582 11426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.812723 11426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:36.812794 11426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:36.845871 11426 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:39893
I20260619 02:44:36.845927 11553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:39893 every 8 connection(s)
I20260619 02:44:36.847155 11426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:36.848026 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11426
I20260619 02:44:36.848212 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:36.848713 8153 external_mini_cluster.cc:1428] Setting key ffedf8cbd45c7f971d41d779ea38c7ec
Jun 19 02:44:36 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837076, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:44:36.868072 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:36.849773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:34755 (local address 127.7.246.126:41591)
0619 02:44:36.849929 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:36.849933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:36.851436 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 02:44:36.853207 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:36.853215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:36.853308 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:36.853427 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:36.855031 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.855985 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.857587 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.858010 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.858148 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:36.862267 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:36.862271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:36.862275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:36.862305 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:36.864803 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:36.865831 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:36.865835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:36.865837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:36.865978 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:36.866480 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:36.866486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:36.866492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:36.866755 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:36.866862 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:36.867633 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 02:44:36.867817 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:44:36.869148 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:36.850436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41591 (local address 127.7.246.66:34755)
0619 02:44:36.851267 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:36.851311 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:36.852945 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:36.853786 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:36.853795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:36.854287 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:36.854859 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:36.854876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.856200 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.856204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:36.857413 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:36.857424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:36.858249 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:36.858258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:36.858506 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:36.859184 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:36.859207 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:36.861982 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:36.864973 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:36.864984 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:36.865014 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:36.865699 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:36.866099 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:36.866107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:36.866112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:36.866339 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:36.866874 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:36.866880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:36.867227 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 02:44:36.867541 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 02:44:36.867937 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":337,"threads_started":1}
I20260619 02:44:36.870697 11554 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41591
I20260619 02:44:36.871045 11554 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:36.871927 11554 heartbeater.cc:511] Master 127.7.246.126:41591 requested a full tablet report, sending...
I20260619 02:44:36.873646 11231 ts_manager.cc:194] Registered new tserver with Master: 2a32a1b33a8a4aba9f2b6913e740d9a4 (127.7.246.66:39893)
I20260619 02:44:36.874663 11231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:34755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:36.892342 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:37.214901 11559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:37.215204 11559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:37.215350 11559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:37.222856 11559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:37.222996 11559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:37.223191 11559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:37.233103 11559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:37.234433 11559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:37.235741 11559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:37.249014 11564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:37.248991 11567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:37.248943 11565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:37.250187 11559 server_base.cc:1034] running on GCE node
I20260619 02:44:37.250809 11559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:37.251737 11559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:37.252979 11559 hybrid_clock.cc:648] HybridClock initialized: now 1781837077252934 us; error 41 us; skew 500 ppm
I20260619 02:44:37.253310 11559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:37.256479 11559 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:37.258116 11559 webserver.cc:460] Webserver started at http://127.7.246.67:46825/ using document root <none> and password file <none>
I20260619 02:44:37.258790 11559 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:37.258893 11559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:37.259151 11559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:37.262096 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "64240cfcb9a549b899468cde8294a8e0"
format_stamp: "Formatted at 2026-06-19 02:44:37 on dist-test-slave-9tkz"
server_key: "add2d436f03fa46bd57c5794cb5b8ad5"
server_key_iv: "020fbce94a816067f40250966a2e1266"
server_key_version: "encryptionkey@0"
I20260619 02:44:37.262852 11559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "64240cfcb9a549b899468cde8294a8e0"
format_stamp: "Formatted at 2026-06-19 02:44:37 on dist-test-slave-9tkz"
server_key: "add2d436f03fa46bd57c5794cb5b8ad5"
server_key_iv: "020fbce94a816067f40250966a2e1266"
server_key_version: "encryptionkey@0"
I20260619 02:44:37.268007 11559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:37.271656 11574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:37.273059 11559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:37.273214 11559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "64240cfcb9a549b899468cde8294a8e0"
format_stamp: "Formatted at 2026-06-19 02:44:37 on dist-test-slave-9tkz"
server_key: "add2d436f03fa46bd57c5794cb5b8ad5"
server_key_iv: "020fbce94a816067f40250966a2e1266"
server_key_version: "encryptionkey@0"
I20260619 02:44:37.273386 11559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:37.285106 11559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:37.289088 11559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:37.289341 11559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:37.290386 11559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:37.291671 11559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:37.291743 11559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:37.291824 11559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:37.291868 11559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:37.328610 11559 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:36975
I20260619 02:44:37.328652 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:36975 every 8 connection(s)
I20260619 02:44:37.329914 11559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:37.331346 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11559
I20260619 02:44:37.331535 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:37.332046 8153 external_mini_cluster.cc:1428] Setting key 87f8fe1cda158e41ff567dbee171a0ff
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:37.350404 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.332602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:49479 (local address 127.7.246.126:41591)
0619 02:44:37.332859 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:37.332865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:37.334493 (+ 1628us) server_negotiation.cc:407] Connection header received
0619 02:44:37.336439 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:37.336443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:37.336505 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:37.336581 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:37.338115 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.338864 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.340319 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.340704 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.340790 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.344095 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:37.344100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:37.344103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:37.344132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:37.347310 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.348097 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.348102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.348106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.348242 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.348667 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.348674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.348679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.348972 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:37.349121 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:37.349825 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 02:44:37.350046 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 02:44:37.351428 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.333257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41591 (local address 127.7.246.67:49479)
0619 02:44:37.334320 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:37.334371 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:37.336202 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:37.336759 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:37.336766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:37.337311 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:37.337976 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:37.337993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.338985 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.338989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:37.340198 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:37.340209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.340880 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.340884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:37.341080 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.341773 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:37.341794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:37.343894 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:37.347446 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:37.347455 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:37.347475 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:37.347972 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:37.348358 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:37.348363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:37.348367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:37.348553 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:37.349102 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:37.349109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:37.349429 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 02:44:37.349739 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 02:44:37.350252 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":518,"threads_started":1}
I20260619 02:44:37.352790 11687 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41591
I20260619 02:44:37.353127 11687 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:37.353937 11687 heartbeater.cc:511] Master 127.7.246.126:41591 requested a full tablet report, sending...
I20260619 02:44:37.355365 11231 ts_manager.cc:194] Registered new tserver with Master: 64240cfcb9a549b899468cde8294a8e0 (127.7.246.67:36975)
I20260619 02:44:37.356179 11231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:49479
I20260619 02:44:37.359923 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:37.382134 11420 heartbeater.cc:503] Master 127.7.246.126:41591 was elected leader, sending a full tablet report...
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:37.408514 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.394431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.7.246.126:41591)
0619 02:44:37.394619 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:37.394628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:37.395108 (+ 480us) server_negotiation.cc:407] Connection header received
0619 02:44:37.395317 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:37.395329 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:37.395391 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:37.395514 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:37.396932 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.397817 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.399158 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.399652 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.399787 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.403604 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:37.403609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:37.403612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:37.403648 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:37.406007 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.406615 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.406621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.406625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.406747 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.407143 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.407150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.407153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.407454 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:37.407695 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:37.407974 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 02:44:37.408186 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:44:37.409091 11231 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56322
I20260619 02:44:37.410869 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11292
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:37.426957 11698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.411984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.7.246.65:36429)
0619 02:44:37.412571 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:37.412578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:37.412601 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:44:37.412652 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:37.412657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:37.412854 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:37.413061 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:37.414739 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.415768 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.416996 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.417650 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.417784 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.421611 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:37.421627 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:37.421649 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:37.421692 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:37.424248 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.424895 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.424905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.424908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.425027 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.425395 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.425400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.425404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.425782 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:37.426075 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:37.426337 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:44:37.426592 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
W20260619 02:44:37.428229 11415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50152
W20260619 02:44:37.429334 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:37.429514 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11292
I20260619 02:44:37.445783 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11426
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:37.462299 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.446656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.7.246.66:39893)
0619 02:44:37.447266 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:37.447273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:37.447300 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:44:37.447367 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:37.447374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:37.447593 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:37.447720 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:37.449086 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.450461 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.451707 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.452272 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.452367 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.456415 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:37.456427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:37.456442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:37.456484 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:37.459489 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.460163 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.460170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.460172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.460302 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.460695 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.460701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.460704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.461044 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:37.461342 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:37.461681 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 02:44:37.461949 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":355,"threads_started":1}
W20260619 02:44:37.463578 11549 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46200
W20260619 02:44:37.464670 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:37.464844 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11426
I20260619 02:44:37.479974 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11559
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837077, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:37.496134 11700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:37.480868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.7.246.67:36975)
0619 02:44:37.481497 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:37.481504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:37.481603 (+ 99us) server_negotiation.cc:407] Connection header received
0619 02:44:37.481677 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:37.481683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:37.481902 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:37.482033 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:37.483282 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.484327 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.485625 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:37.486178 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:37.486333 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:37.490513 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:37.490531 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:37.490547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:37.490591 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:37.493168 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.493993 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.494000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.494003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.494129 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:37.494496 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:37.494502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:37.494505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:37.494896 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:37.495197 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:37.495494 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 02:44:37.495785 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":320,"threads_started":1}
W20260619 02:44:37.497362 11682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36800
W20260619 02:44:37.498540 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11559: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:44:37.498685 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11559
I20260619 02:44:37.513782 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11199
I20260619 02:44:37.515084 11261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:37.615754 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11199
2026-06-19T02:44:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4336 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11704](info): set up 2 sockets
Jun 19 02:44:37 dist-test-slave-9tkz krb5kdc[11704](info): commencing operation
krb5kdc: starting...
W20260619 02:44:38.702112 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:38 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:38.894423 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41597
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:39.224216 11717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:39.224498 11717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:39.224591 11717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:39.231572 11717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:39.231678 11717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:39.231724 11717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:39.231774 11717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:39.231810 11717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:39.241358 11717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:39.242695 11717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:39.243896 11717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:39.255362 11722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:39.255394 11725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:39.255327 11723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:39.256243 11717 server_base.cc:1034] running on GCE node
I20260619 02:44:39.257041 11717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:39.258505 11717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:39.259742 11717 hybrid_clock.cc:648] HybridClock initialized: now 1781837079259701 us; error 50 us; skew 500 ppm
I20260619 02:44:39.260105 11717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:39 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837079, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:39.263208 11717 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:39.264803 11717 webserver.cc:460] Webserver started at http://127.7.246.126:37077/ using document root <none> and password file <none>
I20260619 02:44:39.265458 11717 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:39.265609 11717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:39.265924 11717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:39.268707 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "69371a1ff1f945a7b7c64ed1b271105f"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "7573a6a17b4154a3a980415132dfd308"
server_key_iv: "cf799c712ce67bacb84a290146b88aa3"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.269457 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "69371a1ff1f945a7b7c64ed1b271105f"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "7573a6a17b4154a3a980415132dfd308"
server_key_iv: "cf799c712ce67bacb84a290146b88aa3"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.274442 11717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:39.277781 11732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:39.279088 11717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:39.279260 11717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "69371a1ff1f945a7b7c64ed1b271105f"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "7573a6a17b4154a3a980415132dfd308"
server_key_iv: "cf799c712ce67bacb84a290146b88aa3"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.279452 11717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:39.292213 11717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:39.295984 11717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:39.296268 11717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:39.314280 11717 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41597
I20260619 02:44:39.314311 11783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41597 every 8 connection(s)
I20260619 02:44:39.315660 11717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:39.319680 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:39.322404 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11717
I20260619 02:44:39.322619 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:39.323518 8153 external_mini_cluster.cc:1428] Setting key 5f598c8b516b7e8983aa6b7b18f5f922
I20260619 02:44:39.333091 11784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f: Bootstrap starting.
I20260619 02:44:39.337203 11784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:39.338603 11784 log.cc:824] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:44:39 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:39.342024 11784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f: No bootstrap required, opened a new log
I20260619 02:44:39.346937 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:39.325842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.7.246.126:41597)
0619 02:44:39.326652 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:39.326663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:39.326702 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:39.328302 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:39.328324 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:39.328706 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:39.329031 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:39.331028 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.332283 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.334123 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.334817 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.334946 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:39.339831 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:39.339839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:39.339853 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:39.339885 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:39.342950 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:39.343641 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:39.343647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:39.343656 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:39.343812 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:39.344206 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:39.344210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:39.344212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:39.344738 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:39.345056 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:39.345480 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:44:39.345934 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":388,"threads_started":1}
I20260619 02:44:39.349807 11784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } }
I20260619 02:44:39.350193 11784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:39.350306 11784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69371a1ff1f945a7b7c64ed1b271105f, State: Initialized, Role: FOLLOWER
I20260619 02:44:39.350849 11784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } }
I20260619 02:44:39.351075 11784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:39.351197 11784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:39.351368 11784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:39.353853 11784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } }
I20260619 02:44:39.354385 11784 leader_election.cc:302] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69371a1ff1f945a7b7c64ed1b271105f; no voters:
I20260619 02:44:39.354939 11784 leader_election.cc:288] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:39.355085 11789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:39.355756 11789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [term 1 LEADER]: Becoming Leader. State: Replica: 69371a1ff1f945a7b7c64ed1b271105f, State: Running, Role: LEADER
I20260619 02:44:39.356293 11789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } }
I20260619 02:44:39.357038 11784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:39.359058 11791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69371a1ff1f945a7b7c64ed1b271105f. Latest consensus state: current_term: 1 leader_uuid: "69371a1ff1f945a7b7c64ed1b271105f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } } }
I20260619 02:44:39.359784 11791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:39.359833 11790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69371a1ff1f945a7b7c64ed1b271105f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69371a1ff1f945a7b7c64ed1b271105f" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41597 } } }
I20260619 02:44:39.360216 11790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:39.363286 11797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:39.367472 11797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:39.375669 11797 catalog_manager.cc:1347] Generated new cluster ID: 395ad6932763497684d2acea554405d2
I20260619 02:44:39.375780 11797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:39.391467 11797 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:39.392756 11797 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:39.402258 11797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69371a1ff1f945a7b7c64ed1b271105f: Generated new TSK 0
I20260619 02:44:39.403191 11797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:39.456638 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:39.787083 11810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:39.787376 11810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:39.787469 11810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:39.794697 11810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:39.794809 11810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:39.794931 11810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:39.804469 11810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:39.805819 11810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:39.807121 11810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:39.820175 11818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:39.820258 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:39.820276 11816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:39.820948 11810 server_base.cc:1034] running on GCE node
I20260619 02:44:39.821648 11810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:39.822558 11810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:39.823753 11810 hybrid_clock.cc:648] HybridClock initialized: now 1781837079823715 us; error 37 us; skew 500 ppm
I20260619 02:44:39.824074 11810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:39 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837079, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:39.827217 11810 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:39.828683 11810 webserver.cc:460] Webserver started at http://127.7.246.65:32839/ using document root <none> and password file <none>
I20260619 02:44:39.829347 11810 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:39.829486 11810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:39.829818 11810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:39.832710 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "b00b2c76c52244b8a41ffb46928a7b22"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "d7b65274310615c578b2c2f61c208d11"
server_key_iv: "13a44bebb19fb44d3583e7772887a633"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.833465 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "b00b2c76c52244b8a41ffb46928a7b22"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "d7b65274310615c578b2c2f61c208d11"
server_key_iv: "13a44bebb19fb44d3583e7772887a633"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.838421 11810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 02:44:39.841889 11825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:39.843199 11810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:44:39.843387 11810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "b00b2c76c52244b8a41ffb46928a7b22"
format_stamp: "Formatted at 2026-06-19 02:44:39 on dist-test-slave-9tkz"
server_key: "d7b65274310615c578b2c2f61c208d11"
server_key_iv: "13a44bebb19fb44d3583e7772887a633"
server_key_version: "encryptionkey@0"
I20260619 02:44:39.843595 11810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:39.859251 11810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:39.863425 11810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:39.863713 11810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:39.864629 11810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:39.866047 11810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:39.866147 11810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:39.866300 11810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:39.866371 11810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:39.898672 11810 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44471
I20260619 02:44:39.898720 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44471 every 8 connection(s)
I20260619 02:44:39.899981 11810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:39.906682 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11810
I20260619 02:44:39.906875 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:39.907481 8153 external_mini_cluster.cc:1428] Setting key fd9c785e1b2c3fef5298e8dc360aa73b
Jun 19 02:44:39 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837079, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:39.919567 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:39.902561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:58995 (local address 127.7.246.126:41597)
0619 02:44:39.902980 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:39.902984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:39.903882 (+ 898us) server_negotiation.cc:407] Connection header received
0619 02:44:39.905716 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:39.905721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:39.905808 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:39.905924 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:39.907507 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.908323 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.910015 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.910399 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.910532 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:39.914081 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:39.914088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:39.914100 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:39.914131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:39.916564 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:39.917346 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:39.917351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:39.917355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:39.917469 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:39.917917 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:39.917922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:39.917925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:39.918183 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:39.918342 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:39.918995 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 02:44:39.919202 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260619 02:44:39.920517 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:39.903056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41597 (local address 127.7.246.65:58995)
0619 02:44:39.903725 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:39.903764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:39.905415 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:39.906172 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:39.906181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:39.906702 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:39.907346 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:39.907364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.908504 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.908507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:39.909860 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:39.909872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:39.910614 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:39.910623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:39.910861 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:39.911528 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:39.911551 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:39.913886 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:39.916699 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:39.916707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:39.916723 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:39.917224 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:39.917616 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:39.917621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:39.917625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:39.917801 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:39.918317 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:39.918325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:39.918652 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 02:44:39.918899 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 02:44:39.919424 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
I20260619 02:44:39.922176 11938 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41597
I20260619 02:44:39.922523 11938 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:39.923410 11938 heartbeater.cc:511] Master 127.7.246.126:41597 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:39.925690 11749 ts_manager.cc:194] Registered new tserver with Master: b00b2c76c52244b8a41ffb46928a7b22 (127.7.246.65:44471)
I20260619 02:44:39.927240 11749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:58995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:39.949306 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:40.268482 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:40.268800 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:40.268957 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:40.276410 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:40.276521 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:40.276760 11944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:40.287739 11944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:40.289063 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:40.290489 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:40.303550 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:40.303498 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:40.303532 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:40.304353 11944 server_base.cc:1034] running on GCE node
I20260619 02:44:40.304998 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:40.305997 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:40.307255 11944 hybrid_clock.cc:648] HybridClock initialized: now 1781837080307217 us; error 33 us; skew 500 ppm
I20260619 02:44:40.307576 11944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:40.310559 11944 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:40.311904 11944 webserver.cc:460] Webserver started at http://127.7.246.66:46319/ using document root <none> and password file <none>
I20260619 02:44:40.312527 11944 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:40.312623 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:40.312875 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:40.315698 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "08235bb82d3b429fb73f7c14ca7ff83d"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "67e98b6c037a06f4a4701421312e2a12"
server_key_iv: "89a36106c502d26bc071d900fb17c4ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.316507 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "08235bb82d3b429fb73f7c14ca7ff83d"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "67e98b6c037a06f4a4701421312e2a12"
server_key_iv: "89a36106c502d26bc071d900fb17c4ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.321636 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:44:40.325047 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.326530 11944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:44:40.326716 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "08235bb82d3b429fb73f7c14ca7ff83d"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "67e98b6c037a06f4a4701421312e2a12"
server_key_iv: "89a36106c502d26bc071d900fb17c4ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.326912 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:40.347496 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:40.351830 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:40.352129 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:40.353132 11944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:40.354494 11944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:40.354595 11944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.354770 11944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:40.354847 11944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.388170 11944 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:38909
I20260619 02:44:40.388254 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:38909 every 8 connection(s)
I20260619 02:44:40.389412 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:40.398005 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 11944
I20260619 02:44:40.398166 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:40.398635 8153 external_mini_cluster.cc:1428] Setting key 4dc3a14629502cde8e5a3e0b1b040038
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:40.409093 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.391951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44289 (local address 127.7.246.126:41597)
0619 02:44:40.392145 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:40.392151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:40.393651 (+ 1500us) server_negotiation.cc:407] Connection header received
0619 02:44:40.395619 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:40.395624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:40.395689 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:40.395782 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:40.397234 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.398252 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.399781 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.400248 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.400381 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.403664 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:40.403669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:40.403673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:40.403702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:40.406257 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.406956 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.406967 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.406969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.407084 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.407498 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.407505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.407508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.407768 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:40.407913 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:40.408557 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 02:44:40.408762 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:44:40.410109 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.392505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41597 (local address 127.7.246.66:44289)
0619 02:44:40.393386 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:40.393478 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:40.395262 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:40.396027 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:40.396034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:40.396510 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:40.397082 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:40.397098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.398385 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.398389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:40.399622 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:40.399633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.400464 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.400471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:40.400698 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.401361 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:40.401386 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:40.403451 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:40.406389 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:40.406398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:40.406414 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:40.406828 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:40.407200 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:40.407207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:40.407210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:40.407391 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:40.407888 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:40.407896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:40.408230 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:44:40.408477 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 02:44:40.408989 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":425,"threads_started":1}
I20260619 02:44:40.411598 12072 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41597
I20260619 02:44:40.411892 12072 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:40.412583 12072 heartbeater.cc:511] Master 127.7.246.126:41597 requested a full tablet report, sending...
I20260619 02:44:40.414168 11749 ts_manager.cc:194] Registered new tserver with Master: 08235bb82d3b429fb73f7c14ca7ff83d (127.7.246.66:38909)
I20260619 02:44:40.415493 11749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:44289
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:40.442862 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:40.771806 12077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:40.772091 12077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:40.772183 12077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:40.779438 12077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:40.779569 12077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:40.779685 12077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:40.789615 12077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:40.790912 12077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:40.792167 12077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:40.806041 12082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:40.806002 12085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:40.806016 12083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:40.806800 12077 server_base.cc:1034] running on GCE node
I20260619 02:44:40.807351 12077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:40.808276 12077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:40.809515 12077 hybrid_clock.cc:648] HybridClock initialized: now 1781837080809407 us; error 61 us; skew 500 ppm
I20260619 02:44:40.809864 12077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:40.816028 12077 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:40.817744 12077 webserver.cc:460] Webserver started at http://127.7.246.67:37571/ using document root <none> and password file <none>
I20260619 02:44:40.818526 12077 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:40.818679 12077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:40.819015 12077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:40.822252 12077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "9087e71e0a71430c8fc7e84d6818e45e"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "77cb235035781d1c592d0b0cc7325e7f"
server_key_iv: "f0ef19e62314284d3467c21cb9393a6f"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.823235 12077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "9087e71e0a71430c8fc7e84d6818e45e"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "77cb235035781d1c592d0b0cc7325e7f"
server_key_iv: "f0ef19e62314284d3467c21cb9393a6f"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.828672 12077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:44:40.832304 12092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.833694 12077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:40.833865 12077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "9087e71e0a71430c8fc7e84d6818e45e"
format_stamp: "Formatted at 2026-06-19 02:44:40 on dist-test-slave-9tkz"
server_key: "77cb235035781d1c592d0b0cc7325e7f"
server_key_iv: "f0ef19e62314284d3467c21cb9393a6f"
server_key_version: "encryptionkey@0"
I20260619 02:44:40.834066 12077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:40.859225 12077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:40.863667 12077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:40.863971 12077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:40.864919 12077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:40.866362 12077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:40.866465 12077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.866629 12077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:40.866700 12077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:40.900496 12077 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:35711
I20260619 02:44:40.900547 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:35711 every 8 connection(s)
I20260619 02:44:40.901846 12077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:40.902334 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12077
I20260619 02:44:40.902549 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:40.903239 8153 external_mini_cluster.cc:1428] Setting key 5de1097a1f52373673072126ed187455
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:40.922067 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.904566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:54943 (local address 127.7.246.126:41597)
0619 02:44:40.904756 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:40.904762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:40.906280 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 02:44:40.908128 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:40.908131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:40.908192 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:40.908269 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:40.909772 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.910474 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.911955 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.912376 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.912468 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.916003 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:40.916008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:40.916012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:40.916054 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:40.919285 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.920026 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.920029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.920032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.920160 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.920554 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.920557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.920559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.920807 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:40.920929 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:40.921658 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 02:44:40.921815 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:44:40.923144 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.905112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41597 (local address 127.7.246.67:54943)
0619 02:44:40.906097 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:40.906156 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:40.907877 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:40.908520 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:40.908528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:40.909018 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:40.909625 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:40.909651 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.910606 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.910610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:40.911826 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:40.911837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.912624 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.912633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:40.912846 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.913502 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:40.913525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:40.915802 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:40.919423 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:40.919431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:40.919445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:40.919923 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:40.920248 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:40.920251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:40.920254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:40.920431 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:40.920935 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:40.920940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:40.921261 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 02:44:40.921529 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 02:44:40.922016 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":444,"threads_started":1}
I20260619 02:44:40.924537 12205 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41597
I20260619 02:44:40.924894 12205 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:40.925652 12205 heartbeater.cc:511] Master 127.7.246.126:41597 requested a full tablet report, sending...
I20260619 02:44:40.927079 11749 ts_manager.cc:194] Registered new tserver with Master: 9087e71e0a71430c8fc7e84d6818e45e (127.7.246.67:35711)
I20260619 02:44:40.927824 11749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:54943
I20260619 02:44:40.930119 11938 heartbeater.cc:503] Master 127.7.246.126:41597 was elected leader, sending a full tablet report...
I20260619 02:44:40.931804 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:40.980829 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.965627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.7.246.65:44471)
0619 02:44:40.966285 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:40.966297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:40.966335 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:44:40.966394 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:40.966401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:40.966620 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:40.966788 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:40.968271 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.969354 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.970663 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.971252 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.971392 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.975335 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:40.975348 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:40.975362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:40.975397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:40.978336 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.978962 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.978971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.978974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.979091 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.979452 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.979456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.979458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.979855 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:40.980075 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:40.980295 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 02:44:40.980528 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":281,"threads_started":1}
Jun 19 02:44:40 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:40.998533 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:40.985242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.7.246.126:41597)
0619 02:44:40.985479 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:40.985485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:40.985724 (+ 239us) server_negotiation.cc:407] Connection header received
0619 02:44:40.985921 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:40.985927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:40.986002 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:40.986112 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:40.987321 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.988071 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.989321 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:40.989804 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:40.989932 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:40.993611 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:40.993616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:40.993620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:40.993655 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:40.995933 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.996601 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.996607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.996610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.996745 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:40.997090 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:40.997096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:40.997099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:40.997413 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:40.997683 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:40.997992 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 02:44:40.998221 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 02:44:40.999177 11749 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48114
I20260619 02:44:41.000226 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11810
I20260619 02:44:41.012104 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:41.001095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.7.246.65:44471)
0619 02:44:41.001310 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:41.001316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:41.001493 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:44:41.001678 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:41.001681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:41.001750 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:41.001827 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:41.002883 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.003580 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.004722 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.005133 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.005235 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:41.007433 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:41.007438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:41.007441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:41.007472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:41.010030 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.010584 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.010588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.010590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.010701 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.010993 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.010996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.010998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.011217 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:41.011378 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:41.011601 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:44:41.011751 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:44:41.012684 11933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:41.095403 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11810
I20260619 02:44:41.111143 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11944
Jun 19 02:44:41 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:41.127055 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:41.112012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.7.246.66:38909)
0619 02:44:41.112523 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:41.112530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:41.112554 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:44:41.112611 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:41.112617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:41.112833 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:41.113005 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:41.114337 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.115376 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.116678 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.117271 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.117414 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:41.121522 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:41.121535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:41.121608 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:41.121663 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:41.124655 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.125225 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.125232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.125234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.125350 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.125726 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.125737 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.125739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.126095 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:41.126313 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:41.126560 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 02:44:41.126755 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":266,"threads_started":1}
I20260619 02:44:41.128342 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:41.211503 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11944
I20260619 02:44:41.227164 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12077
Jun 19 02:44:41 dist-test-slave-9tkz krb5kdc[11704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:41.243260 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:41.228191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.7.246.67:35711)
0619 02:44:41.228837 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:41.228843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:41.228870 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:44:41.228926 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:41.228932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:41.229132 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:41.229297 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:41.230568 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.231609 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.232886 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:41.233425 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:41.233583 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:41.237846 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:41.237856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:41.237871 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:41.237904 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:41.240599 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.241313 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.241319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.241322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.241436 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:41.241837 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:41.241840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:41.241842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:41.242165 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:41.242401 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:41.242689 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:44:41.242890 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":365,"threads_started":1}
I20260619 02:44:41.244613 12200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:41.332687 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12077
I20260619 02:44:41.348678 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 11717
I20260619 02:44:41.350008 11779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:41.450340 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 11717
2026-06-19T02:44:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3834 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:41 dist-test-slave-9tkz krb5kdc[12224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:41 dist-test-slave-9tkz krb5kdc[12224](info): set up 2 sockets
Jun 19 02:44:41 dist-test-slave-9tkz krb5kdc[12224](info): commencing operation
krb5kdc: starting...
W20260619 02:44:42.542680 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:42 dist-test-slave-9tkz krb5kdc[12224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:42.753392 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36057
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:43.077479 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:43.077792 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:43.077893 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:43.085201 12237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:43.085345 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:43.085397 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:43.085444 12237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:43.085498 12237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:43.095400 12237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:43.096702 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:43.097995 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:43.109635 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:43.109678 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:43.109715 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:43.110671 12237 server_base.cc:1034] running on GCE node
I20260619 02:44:43.111470 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:43.112936 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:43.114189 12237 hybrid_clock.cc:648] HybridClock initialized: now 1781837083114133 us; error 54 us; skew 500 ppm
I20260619 02:44:43.114544 12237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:43 dist-test-slave-9tkz krb5kdc[12224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837083, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:43.117832 12237 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:43.118943 12237 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:43.119042 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:43.119311 12237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:43.122396 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "b3414a3d15ab406a8f3a1976b0d666bd"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "c8adbd37135232353ffb242bebbdbffe"
server_key_iv: "43d04a136ddc9117d5a2f0260ebbb9c4"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.123142 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "b3414a3d15ab406a8f3a1976b0d666bd"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "c8adbd37135232353ffb242bebbdbffe"
server_key_iv: "43d04a136ddc9117d5a2f0260ebbb9c4"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.128301 12237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:43.131784 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:43.133087 12237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:43.133229 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "b3414a3d15ab406a8f3a1976b0d666bd"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "c8adbd37135232353ffb242bebbdbffe"
server_key_iv: "43d04a136ddc9117d5a2f0260ebbb9c4"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.133428 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:43.178499 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:43.182574 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:43.182803 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:43.201588 12237 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:36057
I20260619 02:44:43.201535 12302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:36057 every 8 connection(s)
I20260619 02:44:43.202553 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:43.203025 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12237
I20260619 02:44:43.203255 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:43.204062 8153 external_mini_cluster.cc:1428] Setting key e287971d3978181f15d10e01c19795d4
I20260619 02:44:43.206995 12303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:44:43 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:43.217890 12303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: Bootstrap starting.
I20260619 02:44:43.221453 12303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:43.222787 12303 log.cc:824] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:43.226017 12303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: No bootstrap required, opened a new log
I20260619 02:44:43.226189 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:43.206137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.7.246.126:36057)
0619 02:44:43.206915 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:43.206926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:43.206965 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:43.208568 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:43.208590 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:43.208979 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:43.209373 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:43.210874 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.212201 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.213455 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.214167 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.214368 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:43.218783 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:43.218791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:43.218805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:43.218839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:43.221846 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:43.222653 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:43.222659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:43.222665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:43.222818 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:43.223214 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:43.223217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:43.223219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:43.223994 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:43.224261 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:43.224715 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 02:44:43.225098 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":371,"threads_started":1}
I20260619 02:44:43.234084 12303 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } }
I20260619 02:44:43.234455 12303 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:43.234609 12303 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3414a3d15ab406a8f3a1976b0d666bd, State: Initialized, Role: FOLLOWER
I20260619 02:44:43.235139 12303 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } }
I20260619 02:44:43.235366 12303 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:43.235498 12303 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:43.235667 12303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:43.238200 12303 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } }
I20260619 02:44:43.238727 12303 leader_election.cc:302] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3414a3d15ab406a8f3a1976b0d666bd; no voters:
I20260619 02:44:43.239275 12303 leader_election.cc:288] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:43.239406 12308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:43.240564 12308 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [term 1 LEADER]: Becoming Leader. State: Replica: b3414a3d15ab406a8f3a1976b0d666bd, State: Running, Role: LEADER
I20260619 02:44:43.241088 12308 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } }
I20260619 02:44:43.241657 12303 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:43.243835 12309 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } } }
I20260619 02:44:43.244199 12309 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:43.244627 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3414a3d15ab406a8f3a1976b0d666bd. Latest consensus state: current_term: 1 leader_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3414a3d15ab406a8f3a1976b0d666bd" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36057 } } }
I20260619 02:44:43.244964 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:43.245326 12317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:43.251509 12317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:43.260454 12317 catalog_manager.cc:1347] Generated new cluster ID: 758eca12d2744fc8863c1bb2c29456cd
I20260619 02:44:43.260578 12317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:43.276460 12317 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:43.277878 12317 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:43.286456 12317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: Generated new TSK 0
I20260619 02:44:43.287358 12317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:43.345719 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:43.676462 12329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:43.676769 12329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:43.676872 12329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:43.684131 12329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:43.684247 12329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:43.684360 12329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:43.694283 12329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:43.695570 12329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:43.696852 12329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:43.709717 12334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:43.709827 12337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:43.709910 12335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:43.710673 12329 server_base.cc:1034] running on GCE node
I20260619 02:44:43.711257 12329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:43.712157 12329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:43.713394 12329 hybrid_clock.cc:648] HybridClock initialized: now 1781837083713335 us; error 57 us; skew 500 ppm
I20260619 02:44:43.713780 12329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:43 dist-test-slave-9tkz krb5kdc[12224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837083, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:43.716848 12329 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:43.718057 12329 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:43.718209 12329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:43.718535 12329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:43.721486 12329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "b9475017979b4b71a5eebffe7a4dd56d"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "96d3ec4e2be809a7562f6e92e8ac574f"
server_key_iv: "4c853fd1c0b38d1d7758965d250658cf"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.722390 12329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "b9475017979b4b71a5eebffe7a4dd56d"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "96d3ec4e2be809a7562f6e92e8ac574f"
server_key_iv: "4c853fd1c0b38d1d7758965d250658cf"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.727947 12329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:43.731640 12343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:43.733001 12329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:43.733184 12329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "b9475017979b4b71a5eebffe7a4dd56d"
format_stamp: "Formatted at 2026-06-19 02:44:43 on dist-test-slave-9tkz"
server_key: "96d3ec4e2be809a7562f6e92e8ac574f"
server_key_iv: "4c853fd1c0b38d1d7758965d250658cf"
server_key_version: "encryptionkey@0"
I20260619 02:44:43.733412 12329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:43.747184 12329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:43.751616 12329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:43.751938 12329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:43.753096 12329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:43.755069 12329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:43.755188 12329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:43.755292 12329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:43.755385 12329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:43.788611 12329 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:43743
I20260619 02:44:43.788707 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:43743 every 8 connection(s)
I20260619 02:44:43.789633 12329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:43.795923 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12329
I20260619 02:44:43.796139 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:43.796937 8153 external_mini_cluster.cc:1428] Setting key bcf9c66401c2238d7c0544b8c2867d65
Jun 19 02:44:43 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837083, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:43.810539 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:43.792267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:60407 (local address 127.7.246.126:36057)
0619 02:44:43.792692 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:43.792697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:43.793794 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 02:44:43.795615 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:43.795621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:43.795684 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:43.795776 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:43.797913 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.798879 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.800459 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.800845 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.800996 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:43.804517 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:43.804522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:43.804525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:43.804596 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:43.807107 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:43.808062 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:43.808066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:43.808069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:43.808182 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:43.808689 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:43.808692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:43.808694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:43.808948 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:43.809070 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:43.810096 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0619 02:44:43.810273 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":241,"threads_started":1}
I20260619 02:44:43.812183 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:43.792903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36057 (local address 127.7.246.65:60407)
0619 02:44:43.793635 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:43.793680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:43.795284 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:43.796077 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:43.796089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:43.796884 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:43.797726 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:43.797746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.799051 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.799061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:43.800291 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:43.800302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:43.801081 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:43.801088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:43.801321 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:43.802125 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:43.802152 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:43.804301 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:43.807244 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:43.807255 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:43.807276 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:43.807927 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:43.808293 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:43.808301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:43.808307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:43.808535 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:43.809090 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:43.809101 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:43.809610 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 02:44:43.809980 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 02:44:43.810594 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":251,"threads_started":1}
I20260619 02:44:43.813956 12456 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36057
I20260619 02:44:43.814254 12456 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:43.814898 12456 heartbeater.cc:511] Master 127.7.246.126:36057 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:43.817058 12268 ts_manager.cc:194] Registered new tserver with Master: b9475017979b4b71a5eebffe7a4dd56d (127.7.246.65:43743)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:44:43.818811 12268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:60407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:43.842660 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:44.168900 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:44.169178 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:44.169337 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:44.176728 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:44.176842 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:44.177038 12462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:44.186935 12462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:44.188328 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:44.189741 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:44.203301 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:44.203239 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:44.203280 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:44.205139 12462 server_base.cc:1034] running on GCE node
I20260619 02:44:44.205809 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:44.206873 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:44.208091 12462 hybrid_clock.cc:648] HybridClock initialized: now 1781837084208033 us; error 56 us; skew 500 ppm
I20260619 02:44:44.208412 12462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:44 dist-test-slave-9tkz krb5kdc[12224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837084, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:44.211622 12462 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:44.212848 12462 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:44.212934 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:44.213284 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:44.216176 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "1898b9c1341f45ccac891bedcbe6555a"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "1c0006045357917980fc1b0532806615"
server_key_iv: "2dd177a9383eea9913fc75e1a1559002"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.216953 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "1898b9c1341f45ccac891bedcbe6555a"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "1c0006045357917980fc1b0532806615"
server_key_iv: "2dd177a9383eea9913fc75e1a1559002"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.222128 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 02:44:44.225519 12476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.226933 12462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:44.227093 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "1898b9c1341f45ccac891bedcbe6555a"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "1c0006045357917980fc1b0532806615"
server_key_iv: "2dd177a9383eea9913fc75e1a1559002"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.227291 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:44.244017 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:44.248216 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:44.248476 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:44.249450 12462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:44.250861 12462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:44.250944 12462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.251103 12462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:44.251199 12462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.284279 12462 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:39009
I20260619 02:44:44.284334 12588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:39009 every 8 connection(s)
I20260619 02:44:44.285164 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:44.292228 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12462
I20260619 02:44:44.292479 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:44.293067 8153 external_mini_cluster.cc:1428] Setting key 362a2c2e797dbb53aad6312f18aa4c3f
Jun 19 02:44:44 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837084, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:44.307217 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.287877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:33871 (local address 127.7.246.126:36057)
0619 02:44:44.288086 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.288092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.289360 (+ 1268us) server_negotiation.cc:407] Connection header received
0619 02:44:44.291423 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.291427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.291497 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.291585 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:44.293851 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.294877 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.296634 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.297060 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.297201 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.300897 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:44.300902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:44.300905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:44.300937 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:44.303563 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.304342 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.304352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.304354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.304468 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.305004 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.305014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.305020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.305461 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:44.305680 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.306442 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.306718 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:44:44.307816 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.288405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36057 (local address 127.7.246.66:33871)
0619 02:44:44.289175 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:44.289230 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:44.291056 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:44.291899 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:44.291933 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:44.292736 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:44.293628 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.293656 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.295101 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.295108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.296473 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.296485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.297295 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.297302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.297527 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.298354 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:44.298377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:44.300654 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:44.303735 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:44.303745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:44.303763 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:44.304206 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:44.304583 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:44.304588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:44.304592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:44.304823 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:44.305651 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:44.305657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:44.306026 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 02:44:44.306260 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 02:44:44.306666 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":309,"threads_started":1}
I20260619 02:44:44.309348 12589 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36057
I20260619 02:44:44.309710 12589 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:44.310375 12589 heartbeater.cc:511] Master 127.7.246.126:36057 requested a full tablet report, sending...
I20260619 02:44:44.311913 12268 ts_manager.cc:194] Registered new tserver with Master: 1898b9c1341f45ccac891bedcbe6555a (127.7.246.66:39009)
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:44:44.312826 12268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:33871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:44.338882 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:44.668488 12594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:44.668784 12594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:44.668882 12594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:44.676390 12594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:44.676594 12594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:44.676785 12594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:44.686956 12594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:44.688356 12594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:44.689661 12594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:44.702302 12599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:44.702340 12602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:44.702283 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:44.703200 12594 server_base.cc:1034] running on GCE node
I20260619 02:44:44.703769 12594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:44.704792 12594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:44.706045 12594 hybrid_clock.cc:648] HybridClock initialized: now 1781837084706005 us; error 51 us; skew 500 ppm
I20260619 02:44:44.706385 12594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:44 dist-test-slave-9tkz krb5kdc[12224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837084, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:44.709782 12594 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:44.711040 12594 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:44.711182 12594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:44.711519 12594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:44.714676 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "9574002ae7f04e1b88c2a8187e300a7e"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "54ef02084569d84877b0268bd96c89a1"
server_key_iv: "5d6056e77c2d96740bfcd40e97881e44"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.715571 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "9574002ae7f04e1b88c2a8187e300a7e"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "54ef02084569d84877b0268bd96c89a1"
server_key_iv: "5d6056e77c2d96740bfcd40e97881e44"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.720935 12594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:44.724802 12608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.726387 12594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:44.726619 12594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "9574002ae7f04e1b88c2a8187e300a7e"
format_stamp: "Formatted at 2026-06-19 02:44:44 on dist-test-slave-9tkz"
server_key: "54ef02084569d84877b0268bd96c89a1"
server_key_iv: "5d6056e77c2d96740bfcd40e97881e44"
server_key_version: "encryptionkey@0"
I20260619 02:44:44.726862 12594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:44.752950 12594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:44.758528 12594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:44.758823 12594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:44.759827 12594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:44.761201 12594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:44.761330 12594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.761458 12594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:44.761588 12594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:44.799850 12594 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:35315
I20260619 02:44:44.799902 12720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:35315 every 8 connection(s)
I20260619 02:44:44.800736 12594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:44.803668 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12594
I20260619 02:44:44.803848 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:44.804437 8153 external_mini_cluster.cc:1428] Setting key 7ec528226f43f2625d9a0ca1f346a38b
Jun 19 02:44:44 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837084, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:44.821699 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.803170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:55989 (local address 127.7.246.126:36057)
0619 02:44:44.803338 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.803343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.804729 (+ 1386us) server_negotiation.cc:407] Connection header received
0619 02:44:44.806947 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.806953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.807048 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.807152 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:44.808988 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.810101 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.811671 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.812109 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.812215 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.815631 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:44.815636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:44.815640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:44.815674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:44.818301 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.819184 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.819192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.819195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.819377 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.819884 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.819889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.819893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.820175 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:44.820357 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.821104 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.821316 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:44:44.821713 12456 heartbeater.cc:503] Master 127.7.246.126:36057 was elected leader, sending a full tablet report...
I20260619 02:44:44.823092 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.803735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36057 (local address 127.7.246.67:55989)
0619 02:44:44.804546 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:44.804609 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:44.806674 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:44.807419 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:44.807432 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:44.807970 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:44.808734 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.808762 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.810253 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.810257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.811511 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.811522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.812302 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.812306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.812490 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.813201 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:44.813225 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:44.815416 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:44.818439 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:44.818448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:44.818468 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:44.819048 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:44.819498 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:44.819502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:44.819504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:44.819767 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:44.820330 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:44.820338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:44.820695 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 02:44:44.820994 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 02:44:44.821579 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260619 02:44:44.824436 12721 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36057
I20260619 02:44:44.824749 12721 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:44.825404 12721 heartbeater.cc:511] Master 127.7.246.126:36057 requested a full tablet report, sending...
I20260619 02:44:44.826931 12268 ts_manager.cc:194] Registered new tserver with Master: 9574002ae7f04e1b88c2a8187e300a7e (127.7.246.67:35315)
I20260619 02:44:44.827740 12268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:55989
I20260619 02:44:44.832347 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:44.849803 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.836934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.7.246.126:36057)
0619 02:44:44.837124 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.837129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.837610 (+ 481us) server_negotiation.cc:407] Connection header received
0619 02:44:44.837815 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.837819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.837883 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.837986 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:44.839428 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.840157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.841443 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.842006 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.842136 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.844395 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:44.844401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:44.844410 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:44.844444 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:44.847069 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.847785 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.847792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.847795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.847924 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:44.848281 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:44.848288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:44.848294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:44.848632 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:44.848883 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.849227 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.849437 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:44:44.854269 12268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:44.861090 12268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:44.884483 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.873679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35548)
0619 02:44:44.874313 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:44.874344 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:44.874475 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:44.875843 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:44.875853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:44.875892 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:44.876317 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.876334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.878663 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.878668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.881272 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.881291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.882916 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.882922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.883118 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.883172 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:44:44.883317 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 02:44:44.883543 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":101,"thread_start_us":333,"threads_started":1}
I20260619 02:44:44.884634 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.873761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.7.246.65:43743)
0619 02:44:44.875038 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.875046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.875075 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:44:44.875197 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.875209 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.875471 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.875678 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:44.876718 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.878507 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.881451 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.882712 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.882848 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.883084 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.884132 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.884296 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":921,"threads_started":1}
I20260619 02:44:44.884714 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.874980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:35315 (local address 127.0.0.1:33306)
0619 02:44:44.875491 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:44.875511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:44.875624 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:44.877586 (+ 1962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:44.877591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:44.877616 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:44.878249 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.878267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.879720 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.879729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.881743 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.881757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.883456 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.883462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.883611 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.883639 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:44:44.883718 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:44:44.883839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":1,"thread_start_us":280,"threads_started":1}
I20260619 02:44:44.885089 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.875608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39009 (local address 127.0.0.1:33326)
0619 02:44:44.876267 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:44.876293 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:44.876397 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:44.877812 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:44.877817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:44.877843 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:44.878267 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.878276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.880187 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.880192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.882330 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:44.882344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.884131 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.884138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:44.884305 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.884339 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:44:44.884403 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:44:44.884554 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":40,"thread_start_us":782,"threads_started":1}
I20260619 02:44:44.885854 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.875324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.7.246.67:35315)
0619 02:44:44.876940 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.876952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.876988 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:44:44.877079 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.877088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.877350 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.877953 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:44.878465 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.879562 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.881902 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.883244 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.884134 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.884361 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.884430 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.885501 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":1296,"threads_started":1}
I20260619 02:44:44.885878 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:44.876202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.7.246.66:39009)
0619 02:44:44.877207 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:44.877215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:44.877244 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:44:44.877338 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:44.877345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:44.877607 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:44.877824 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:44.878478 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.880031 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.882502 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:44.883869 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:44.885078 (+ 1209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:44.885282 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:44.885348 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:44:44.885512 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":830,"threads_started":1}
I20260619 02:44:44.888756 12391 tablet_service.cc:1467] Processing CreateTablet for tablet 69263a86cbe94a21af9841ee0d87288f (DEFAULT_TABLE table=test-table [id=df4b58eeb7204104947d86f5c813a7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:44.889370 12656 tablet_service.cc:1467] Processing CreateTablet for tablet 69263a86cbe94a21af9841ee0d87288f (DEFAULT_TABLE table=test-table [id=df4b58eeb7204104947d86f5c813a7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:44.889369 12524 tablet_service.cc:1467] Processing CreateTablet for tablet 69263a86cbe94a21af9841ee0d87288f (DEFAULT_TABLE table=test-table [id=df4b58eeb7204104947d86f5c813a7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:44.890427 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69263a86cbe94a21af9841ee0d87288f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:44.890686 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69263a86cbe94a21af9841ee0d87288f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:44.890734 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69263a86cbe94a21af9841ee0d87288f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:44.903034 12739 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Bootstrap starting.
I20260619 02:44:44.905089 12740 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Bootstrap starting.
I20260619 02:44:44.906250 12741 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Bootstrap starting.
I20260619 02:44:44.908622 12740 tablet_bootstrap.cc:654] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:44.909029 12739 tablet_bootstrap.cc:654] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:44.909771 12740 log.cc:824] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:44.909945 12741 tablet_bootstrap.cc:654] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:44.910216 12739 log.cc:824] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:44.911039 12741 log.cc:824] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:44.912942 12740 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: No bootstrap required, opened a new log
I20260619 02:44:44.913201 12739 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: No bootstrap required, opened a new log
I20260619 02:44:44.913233 12740 ts_tablet_manager.cc:1397] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 02:44:44.913472 12739 ts_tablet_manager.cc:1397] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 02:44:44.913915 12741 tablet_bootstrap.cc:492] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: No bootstrap required, opened a new log
I20260619 02:44:44.914175 12741 ts_tablet_manager.cc:1397] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260619 02:44:44.921408 12740 raft_consensus.cc:348] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.921908 12740 raft_consensus.cc:374] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:44.922014 12740 raft_consensus.cc:729] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9475017979b4b71a5eebffe7a4dd56d, State: Initialized, Role: FOLLOWER
I20260619 02:44:44.922684 12740 consensus_queue.cc:260] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.924475 12740 ts_tablet_manager.cc:1428] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 02:44:44.924569 12741 raft_consensus.cc:348] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.925033 12741 raft_consensus.cc:374] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:44.925143 12741 raft_consensus.cc:729] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1898b9c1341f45ccac891bedcbe6555a, State: Initialized, Role: FOLLOWER
I20260619 02:44:44.925133 12739 raft_consensus.cc:348] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.925617 12739 raft_consensus.cc:374] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:44.925724 12739 raft_consensus.cc:729] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9574002ae7f04e1b88c2a8187e300a7e, State: Initialized, Role: FOLLOWER
I20260619 02:44:44.925868 12741 consensus_queue.cc:260] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.926432 12739 consensus_queue.cc:260] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:44.929400 12721 heartbeater.cc:503] Master 127.7.246.126:36057 was elected leader, sending a full tablet report...
I20260619 02:44:44.930122 12739 ts_tablet_manager.cc:1428] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260619 02:44:44.930893 12589 heartbeater.cc:503] Master 127.7.246.126:36057 was elected leader, sending a full tablet report...
I20260619 02:44:44.932884 12741 ts_tablet_manager.cc:1428] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
W20260619 02:44:45.037372 12590 tablet.cc:2369] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:44:45.042070 12457 tablet.cc:2369] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:44:45.052343 12722 tablet.cc:2369] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:44:45.258527 12747 raft_consensus.cc:482] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:45.258814 12747 raft_consensus.cc:504] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:45.260419 12747 leader_election.cc:288] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9475017979b4b71a5eebffe7a4dd56d (127.7.246.65:43743), 1898b9c1341f45ccac891bedcbe6555a (127.7.246.66:39009)
I20260619 02:44:45.267325 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.260806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.7.246.67:44421)
0619 02:44:45.261011 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.261032 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:45.261166 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:45.261573 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:45.261579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:45.261608 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:45.261835 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:45.261852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.263192 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.263196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:45.265168 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:45.265184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.266715 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.266720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:45.266863 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.266888 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:44:45.266993 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:44:45.267104 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":30}
I20260619 02:44:45.267508 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.260947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:44421 (local address 127.7.246.65:43743)
0619 02:44:45.261131 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.261135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.261154 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:44:45.261318 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.261323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.261402 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.261521 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:45.262023 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.263024 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.265409 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.266502 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.266705 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.266785 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.267079 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.267230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:44:45.268608 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.261263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39009 (local address 127.7.246.67:51597)
0619 02:44:45.262059 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.262073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:45.262156 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:45.262455 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:45.262458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:45.262475 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:44:45.262743 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:45.262750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.263839 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.263845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:45.265818 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:45.265829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.267878 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.267893 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:45.268022 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.268047 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:44:45.268131 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:44:45.268224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":4,"thread_start_us":512,"threads_started":1}
I20260619 02:44:45.268621 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.261467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:51597 (local address 127.7.246.66:39009)
0619 02:44:45.261752 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.261760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.262178 (+ 418us) server_negotiation.cc:407] Connection header received
0619 02:44:45.262260 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.262265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.262341 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.262438 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:44:45.262884 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.263675 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.265996 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.267397 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.267608 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.267700 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.268195 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.268331 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 02:44:45.268610 12411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69263a86cbe94a21af9841ee0d87288f" candidate_uuid: "9574002ae7f04e1b88c2a8187e300a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9475017979b4b71a5eebffe7a4dd56d" is_pre_election: true
I20260619 02:44:45.269225 12411 raft_consensus.cc:2454] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9574002ae7f04e1b88c2a8187e300a7e in term 0.
I20260619 02:44:45.269663 12544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69263a86cbe94a21af9841ee0d87288f" candidate_uuid: "9574002ae7f04e1b88c2a8187e300a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1898b9c1341f45ccac891bedcbe6555a" is_pre_election: true
I20260619 02:44:45.270022 12610 leader_election.cc:302] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9574002ae7f04e1b88c2a8187e300a7e, b9475017979b4b71a5eebffe7a4dd56d; no voters:
I20260619 02:44:45.270138 12544 raft_consensus.cc:2454] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9574002ae7f04e1b88c2a8187e300a7e in term 0.
I20260619 02:44:45.270413 12747 raft_consensus.cc:2781] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:45.270567 12747 raft_consensus.cc:482] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:45.270670 12747 raft_consensus.cc:3037] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:45.272796 12747 raft_consensus.cc:504] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:45.273386 12747 leader_election.cc:288] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [CANDIDATE]: Term 1 election: Requested vote from peers b9475017979b4b71a5eebffe7a4dd56d (127.7.246.65:43743), 1898b9c1341f45ccac891bedcbe6555a (127.7.246.66:39009)
I20260619 02:44:45.273974 12411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69263a86cbe94a21af9841ee0d87288f" candidate_uuid: "9574002ae7f04e1b88c2a8187e300a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9475017979b4b71a5eebffe7a4dd56d"
I20260619 02:44:45.274063 12544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69263a86cbe94a21af9841ee0d87288f" candidate_uuid: "9574002ae7f04e1b88c2a8187e300a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1898b9c1341f45ccac891bedcbe6555a"
I20260619 02:44:45.274174 12411 raft_consensus.cc:3037] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:45.274317 12544 raft_consensus.cc:3037] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:45.276481 12411 raft_consensus.cc:2454] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9574002ae7f04e1b88c2a8187e300a7e in term 1.
I20260619 02:44:45.276554 12544 raft_consensus.cc:2454] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9574002ae7f04e1b88c2a8187e300a7e in term 1.
I20260619 02:44:45.276950 12610 leader_election.cc:302] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9574002ae7f04e1b88c2a8187e300a7e, b9475017979b4b71a5eebffe7a4dd56d; no voters:
I20260619 02:44:45.277251 12747 raft_consensus.cc:2781] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:45.277751 12747 raft_consensus.cc:686] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 1 LEADER]: Becoming Leader. State: Replica: 9574002ae7f04e1b88c2a8187e300a7e, State: Running, Role: LEADER
I20260619 02:44:45.278332 12747 consensus_queue.cc:237] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } }
I20260619 02:44:45.282788 12268 catalog_manager.cc:5495] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9574002ae7f04e1b88c2a8187e300a7e (127.7.246.67). New cstate: current_term: 1 leader_uuid: "9574002ae7f04e1b88c2a8187e300a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9574002ae7f04e1b88c2a8187e300a7e" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 35315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:45.343479 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.338212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.7.246.67:35315)
0619 02:44:45.338397 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.338402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.338488 (+ 86us) server_negotiation.cc:407] Connection header received
0619 02:44:45.338649 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.338653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.338741 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.338872 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:44:45.339197 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.339924 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.341262 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.341768 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.341881 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.342423 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:44:45.342966 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:44:45.343129 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.343208 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.343276 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:44:45.351719 12411 raft_consensus.cc:1261] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 1 FOLLOWER]: Refusing update from remote peer 9574002ae7f04e1b88c2a8187e300a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:45.351719 12544 raft_consensus.cc:1261] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 1 FOLLOWER]: Refusing update from remote peer 9574002ae7f04e1b88c2a8187e300a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:45.352749 12749 consensus_queue.cc:1035] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9475017979b4b71a5eebffe7a4dd56d" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:45.353150 12747 consensus_queue.cc:1035] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1898b9c1341f45ccac891bedcbe6555a" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:45.364966 12752 mvcc.cc:205] Tried to move back new op lower bound from 7298404701590204416 to 7298404701303848960. Current Snapshot: MvccSnapshot[applied={T|T < 7298404701590204416}]
I20260619 02:44:45.380813 12268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:45.381126 12268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:45.384847 12268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3414a3d15ab406a8f3a1976b0d666bd: Sending DeleteTablet for 3 replicas of tablet 69263a86cbe94a21af9841ee0d87288f
I20260619 02:44:45.386446 12656 tablet_service.cc:1514] Processing DeleteTablet for tablet 69263a86cbe94a21af9841ee0d87288f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:45 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:33306
I20260619 02:44:45.386399 12391 tablet_service.cc:1514] Processing DeleteTablet for tablet 69263a86cbe94a21af9841ee0d87288f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:45 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35548
I20260619 02:44:45.386502 12524 tablet_service.cc:1514] Processing DeleteTablet for tablet 69263a86cbe94a21af9841ee0d87288f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:45 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:33326
I20260619 02:44:45.388059 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12329
I20260619 02:44:45.399793 12763 tablet_replica.cc:330] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: stopping tablet replica
I20260619 02:44:45.400447 12763 raft_consensus.cc:2229] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:45.402347 12764 tablet_replica.cc:330] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: stopping tablet replica
I20260619 02:44:45.402976 12764 raft_consensus.cc:2229] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:45.403683 12764 raft_consensus.cc:2258] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:45.401432 12763 raft_consensus.cc:2258] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:45.406647 12764 ts_tablet_manager.cc:1905] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:45.410538 12762 tablet_replica.cc:330] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: stopping tablet replica
I20260619 02:44:45.411033 12762 raft_consensus.cc:2229] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:45.411692 12762 raft_consensus.cc:2258] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:45.413194 12763 ts_tablet_manager.cc:1905] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 02:44:45 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:45.417114 12764 ts_tablet_manager.cc:1918] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:45.417326 12764 log.cc:1196] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/69263a86cbe94a21af9841ee0d87288f
I20260619 02:44:45.417955 12764 ts_tablet_manager.cc:1939] T 69263a86cbe94a21af9841ee0d87288f P 1898b9c1341f45ccac891bedcbe6555a: Deleting consensus metadata
I20260619 02:44:45.418215 12762 ts_tablet_manager.cc:1905] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:45.419914 12253 catalog_manager.cc:4841] TS 1898b9c1341f45ccac891bedcbe6555a (127.7.246.66:39009): tablet 69263a86cbe94a21af9841ee0d87288f (table test-table [id=df4b58eeb7204104947d86f5c813a7d2]) successfully deleted
I20260619 02:44:45.423462 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.389590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.7.246.65:43743)
0619 02:44:45.389838 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.389849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.402593 (+ 12744us) server_negotiation.cc:407] Connection header received
0619 02:44:45.402898 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.402904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.403041 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.403120 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:45.404956 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.406228 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.408133 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.408835 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.408939 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.415007 (+ 6068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:45.415026 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:45.415046 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:45.415106 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:45.419270 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.420329 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.420341 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.420349 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.420553 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.421113 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.421121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.421124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.421630 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:45.421957 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.422643 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.422946 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:44:45.425447 12451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:45.426142 12763 ts_tablet_manager.cc:1918] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:45.426357 12763 log.cc:1196] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/69263a86cbe94a21af9841ee0d87288f
I20260619 02:44:45.427083 12763 ts_tablet_manager.cc:1939] T 69263a86cbe94a21af9841ee0d87288f P 9574002ae7f04e1b88c2a8187e300a7e: Deleting consensus metadata
I20260619 02:44:45.429337 12253 catalog_manager.cc:4841] TS 9574002ae7f04e1b88c2a8187e300a7e (127.7.246.67:35315): tablet 69263a86cbe94a21af9841ee0d87288f (table test-table [id=df4b58eeb7204104947d86f5c813a7d2]) successfully deleted
I20260619 02:44:45.517006 12762 ts_tablet_manager.cc:1918] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:45.517277 12762 log.cc:1196] T 69263a86cbe94a21af9841ee0d87288f P b9475017979b4b71a5eebffe7a4dd56d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/69263a86cbe94a21af9841ee0d87288f
I20260619 02:44:45.517619 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12329
W20260619 02:44:45.540421 12253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:43743 (error 108)
I20260619 02:44:45.540624 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12462
I20260619 02:44:45.543044 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.542167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35580)
0619 02:44:45.542627 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.542810 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
W20260619 02:44:45.543464 12253 catalog_manager.cc:4584] TS b9475017979b4b71a5eebffe7a4dd56d (127.7.246.65:43743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69263a86cbe94a21af9841ee0d87288f: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Jun 19 02:44:45 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:45.557168 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.541986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.7.246.66:39009)
0619 02:44:45.542203 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.542209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.542230 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:44:45.542300 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.542308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.542415 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.542529 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:45.544196 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.544893 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.546311 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.546834 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.546980 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.551117 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:45.551132 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:45.551149 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:45.551186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:45.554146 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.555005 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.555013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.555016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.555163 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.555574 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.555584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.555586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.555969 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:45.556268 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.556601 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.556829 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 02:44:45.557928 12584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:45.597265 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.596666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35592)
0619 02:44:45.596934 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.597058 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 02:44:45.598479 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.598045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35608)
0619 02:44:45.598218 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.598307 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 02:44:45.648643 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12462
I20260619 02:44:45.659497 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.658928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35618)
0619 02:44:45.659201 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.659315 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 02:44:45.660693 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.660277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35626)
0619 02:44:45.660463 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.660556 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:44:45.671118 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12594
Jun 19 02:44:45 dist-test-slave-9tkz krb5kdc[12224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:45.687907 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.671951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.7.246.67:35315)
0619 02:44:45.672141 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:45.672149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:45.672172 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:44:45.672378 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:45.672384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:45.672499 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:45.672620 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:45.673990 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.674834 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.676146 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:45.676615 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:45.676756 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:45.681370 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:45.681383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:45.681397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:45.681443 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:45.684812 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.685617 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.685630 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.685636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.685867 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:45.686291 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:45.686297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:45.686300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:45.686680 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:45.687006 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:45.687290 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 02:44:45.687510 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:44:45.688658 12716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:45.741492 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.741002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35640)
0619 02:44:45.741231 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.741345 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 02:44:45.742673 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.742273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35648)
0619 02:44:45.742480 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.742547 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 02:44:45.812132 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12594
I20260619 02:44:45.830825 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12237
I20260619 02:44:45.831907 12298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:45.937372 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:45.936743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43743 (local address 127.0.0.1:35658)
0619 02:44:45.937016 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:45.937144 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 02:44:45.937858 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12237
2026-06-19T02:44:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4487 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:45Z Disabled control of system clock
I20260619 02:44:46.004455 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43965
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:44:46.321676 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:46.322111 12775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:46.322286 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:46.329684 12775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:46.329810 12775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:46.329846 12775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:46.329893 12775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:46.329936 12775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:46.339179 12775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:46.340463 12775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:46.342353 12775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:46.354205 12780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:46.354262 12783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:46.354387 12781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:46.354987 12775 server_base.cc:1034] running on GCE node
I20260619 02:44:46.355875 12775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:46.357366 12775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:46.358592 12775 hybrid_clock.cc:648] HybridClock initialized: now 1781837086358543 us; error 51 us; skew 500 ppm
I20260619 02:44:46.358952 12775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:46.364837 12775 webserver.cc:460] Webserver started at http://127.7.246.126:33169/ using document root <none> and password file <none>
I20260619 02:44:46.365615 12775 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:46.365758 12775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:46.366062 12775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:46.368958 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "51ed5ab583d94d30aefc0e1cdddbfce6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "83054185f0fc1ce22373358fa61547fc"
server_key_iv: "d0dd90425828167564c2a3ba7c7c3c23"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.369802 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "51ed5ab583d94d30aefc0e1cdddbfce6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "83054185f0fc1ce22373358fa61547fc"
server_key_iv: "d0dd90425828167564c2a3ba7c7c3c23"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.374818 12775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 02:44:46.378247 12789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:46.379753 12775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:46.379923 12775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "51ed5ab583d94d30aefc0e1cdddbfce6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "83054185f0fc1ce22373358fa61547fc"
server_key_iv: "d0dd90425828167564c2a3ba7c7c3c23"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.380132 12775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:46.385963 12775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:46.386704 12775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:46.386984 12775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:46.406754 12775 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43965
I20260619 02:44:46.406770 12840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43965 every 8 connection(s)
I20260619 02:44:46.408277 12775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:46.411947 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:46.413161 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12775
I20260619 02:44:46.413391 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:46.414108 8153 external_mini_cluster.cc:1428] Setting key a92f6bafdad636c809591fa58c3f6dd6
I20260619 02:44:46.422824 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:46.416378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.7.246.126:43965)
0619 02:44:46.417488 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:46.417500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:46.417540 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:44:46.419184 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:46.419207 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:46.419634 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:46.419971 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:46.421082 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:46.421089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:46.421102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:46.421119 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:46.421190 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:46.421227 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:46.421420 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:46.421626 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 02:44:46.421925 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":402,"threads_started":1}
I20260619 02:44:46.423810 12841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: Bootstrap starting.
I20260619 02:44:46.426779 12841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:46.427925 12841 log.cc:824] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:46.430460 12841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: No bootstrap required, opened a new log
I20260619 02:44:46.437386 12841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } }
I20260619 02:44:46.437836 12841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:46.437943 12841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ed5ab583d94d30aefc0e1cdddbfce6, State: Initialized, Role: FOLLOWER
I20260619 02:44:46.438591 12841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } }
I20260619 02:44:46.438838 12841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:46.438980 12841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:46.439155 12841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:46.441509 12841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } }
I20260619 02:44:46.442135 12841 leader_election.cc:302] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ed5ab583d94d30aefc0e1cdddbfce6; no voters:
I20260619 02:44:46.442734 12841 leader_election.cc:288] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:46.442875 12846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:46.443608 12846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [term 1 LEADER]: Becoming Leader. State: Replica: 51ed5ab583d94d30aefc0e1cdddbfce6, State: Running, Role: LEADER
I20260619 02:44:46.444115 12846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } }
I20260619 02:44:46.444888 12841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:46.446954 12847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } } }
I20260619 02:44:46.447312 12847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:46.448804 12848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ed5ab583d94d30aefc0e1cdddbfce6. Latest consensus state: current_term: 1 leader_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ed5ab583d94d30aefc0e1cdddbfce6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43965 } } }
I20260619 02:44:46.449123 12848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:46.449378 12851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:46.454977 12851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:46.465917 12851 catalog_manager.cc:1347] Generated new cluster ID: 1ea94d2f9b6d46e4aa3a9a48711ca0ad
I20260619 02:44:46.466048 12851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:46.493662 12851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:46.494103 12851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:46.518036 12851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: Generated new TSK 0
I20260619 02:44:46.519194 12851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:44:46.539197 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--builtin_ntp_servers=127.7.246.84:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:44:46.876955 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:46.877292 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:46.877449 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:46.885519 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:46.885713 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:46.885891 12865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:46.895301 12865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:46.896649 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:46.898518 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:46.911643 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:46.911585 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:46.911675 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:46.912739 12865 server_base.cc:1034] running on GCE node
I20260619 02:44:46.913334 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:46.914331 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:46.915555 12865 hybrid_clock.cc:648] HybridClock initialized: now 1781837086915505 us; error 47 us; skew 500 ppm
I20260619 02:44:46.915869 12865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:46.918067 12865 webserver.cc:460] Webserver started at http://127.7.246.65:36369/ using document root <none> and password file <none>
I20260619 02:44:46.918766 12865 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:46.918895 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:46.919202 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:46.922169 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "adc591455aae4e018ee55d7cc99459c6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "b0850f48de188c8bfb1e61c5e448db6b"
server_key_iv: "62867919854e493076dd0d6fe930b4c5"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.923071 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "adc591455aae4e018ee55d7cc99459c6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "b0850f48de188c8bfb1e61c5e448db6b"
server_key_iv: "62867919854e493076dd0d6fe930b4c5"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.928205 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 02:44:46.931663 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:46.932976 12865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:44:46.933157 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "adc591455aae4e018ee55d7cc99459c6"
format_stamp: "Formatted at 2026-06-19 02:44:46 on dist-test-slave-9tkz"
server_key: "b0850f48de188c8bfb1e61c5e448db6b"
server_key_iv: "62867919854e493076dd0d6fe930b4c5"
server_key_version: "encryptionkey@0"
I20260619 02:44:46.933389 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:46.939289 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:46.940075 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:46.940346 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:46.941265 12865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:46.942699 12865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:46.942821 12865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:46.942950 12865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:46.943034 12865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:46.976259 12865 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:43449
I20260619 02:44:46.976306 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:43449 every 8 connection(s)
I20260619 02:44:46.977869 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:46.981719 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12865
I20260619 02:44:46.981923 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:46.982720 8153 external_mini_cluster.cc:1428] Setting key 9aaf2562f432a6a1d1344befce62f141
I20260619 02:44:46.985515 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--builtin_ntp_servers=127.7.246.84:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:44:46.987690 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:46.980580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:51679 (local address 127.7.246.126:43965)
0619 02:44:46.981026 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:46.981030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:46.982397 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 02:44:46.984673 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:46.984679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:46.984738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:46.984833 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:46.986735 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:46.986743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:46.986748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:46.986769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:46.986825 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:46.986839 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:46.987034 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:46.987256 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 02:44:46.987329 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":253,"threads_started":1}
I20260619 02:44:46.988711 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:46.981051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43965 (local address 127.7.246.65:51679)
0619 02:44:46.982226 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:46.982273 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:46.984381 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:46.985209 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:46.985220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:46.985341 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:46.986358 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:46.986360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:46.986443 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:46.986540 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:46.986994 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:46.987008 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:46.987019 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:44:46.987341 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 02:44:46.987646 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":307,"threads_started":1}
I20260619 02:44:46.990398 12992 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43965
I20260619 02:44:46.990705 12992 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:46.991273 12992 heartbeater.cc:511] Master 127.7.246.126:43965 requested a full tablet report, sending...
I20260619 02:44:46.993472 12806 ts_manager.cc:194] Registered new tserver with Master: adc591455aae4e018ee55d7cc99459c6 (127.7.246.65:43449)
W20260619 02:44:47.308082 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:47.308368 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:47.308458 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:47.315644 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:47.315752 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:47.315862 12996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:47.324990 12996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:47.326282 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:47.327967 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:47.341039 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:47.341151 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:47.341154 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:47.342314 12996 server_base.cc:1034] running on GCE node
I20260619 02:44:47.342936 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:47.343894 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:47.345118 12996 hybrid_clock.cc:648] HybridClock initialized: now 1781837087345061 us; error 53 us; skew 500 ppm
I20260619 02:44:47.345443 12996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:47.347900 12996 webserver.cc:460] Webserver started at http://127.7.246.66:38455/ using document root <none> and password file <none>
I20260619 02:44:47.348613 12996 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:47.348748 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:47.349064 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:47.352101 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "ce5a9bf52f0c43fe88d91323bd0bb499"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "c7b9a5a6e8534bfc6e8d0dea26a9ea8e"
server_key_iv: "71a79e97cb6729103b8dcd7392cbfecf"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.352902 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "ce5a9bf52f0c43fe88d91323bd0bb499"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "c7b9a5a6e8534bfc6e8d0dea26a9ea8e"
server_key_iv: "71a79e97cb6729103b8dcd7392cbfecf"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.358251 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:44:47.361752 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.363078 12996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:47.363251 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "ce5a9bf52f0c43fe88d91323bd0bb499"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "c7b9a5a6e8534bfc6e8d0dea26a9ea8e"
server_key_iv: "71a79e97cb6729103b8dcd7392cbfecf"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.363490 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:47.369294 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:47.370150 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:47.370424 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:47.371325 12996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:47.372684 12996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:47.372797 12996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.372961 12996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:47.373045 12996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.405241 12996 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:37237
I20260619 02:44:47.405287 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:37237 every 8 connection(s)
I20260619 02:44:47.406920 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:47.415227 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.409615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:42167 (local address 127.7.246.126:43965)
0619 02:44:47.409785 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.409789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.411309 (+ 1520us) server_negotiation.cc:407] Connection header received
0619 02:44:47.412974 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.412979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.413047 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.413151 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.414499 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.414505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.414509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.414529 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.414557 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.414568 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.414690 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.414867 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.414949 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:44:47.416195 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.410229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43965 (local address 127.7.246.66:42167)
0619 02:44:47.411121 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:47.411161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:47.412685 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:47.413369 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:47.413377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:47.413465 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:47.414223 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:47.414226 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:47.414261 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:47.414340 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:47.414663 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:47.414673 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:47.414680 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:44:47.414965 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 02:44:47.415223 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":287,"threads_started":1}
I20260619 02:44:47.416440 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 12996
I20260619 02:44:47.416628 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:47.417110 8153 external_mini_cluster.cc:1428] Setting key ed938f8cc27961d644a727c00c83c0a4
I20260619 02:44:47.417689 13123 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43965
I20260619 02:44:47.417932 13123 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:47.418622 13123 heartbeater.cc:511] Master 127.7.246.126:43965 requested a full tablet report, sending...
I20260619 02:44:47.420051 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--builtin_ntp_servers=127.7.246.84:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:44:47.420121 12806 ts_manager.cc:194] Registered new tserver with Master: ce5a9bf52f0c43fe88d91323bd0bb499 (127.7.246.66:37237)
W20260619 02:44:47.747092 13126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:47.747443 13126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:47.747611 13126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:47.755256 13126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:47.755431 13126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:47.755620 13126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:47.765004 13126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:47.766784 13126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:47.769217 13126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:47.782366 13132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:47.782637 13131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:47.783066 13134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:47.783598 13126 server_base.cc:1034] running on GCE node
I20260619 02:44:47.784266 13126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:47.785257 13126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:47.786484 13126 hybrid_clock.cc:648] HybridClock initialized: now 1781837087786405 us; error 72 us; skew 500 ppm
I20260619 02:44:47.786815 13126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:47.789188 13126 webserver.cc:460] Webserver started at http://127.7.246.67:43461/ using document root <none> and password file <none>
I20260619 02:44:47.789951 13126 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:47.790088 13126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:47.790414 13126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:47.793264 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "8f26c629715947beadbb28dcfc6a3a40"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "1bc24a21c16b3419d81c7b46d844e3f3"
server_key_iv: "5daca7313e039e64f10fb12951f82d93"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.794080 13126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "8f26c629715947beadbb28dcfc6a3a40"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "1bc24a21c16b3419d81c7b46d844e3f3"
server_key_iv: "5daca7313e039e64f10fb12951f82d93"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.799130 13126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:47.802512 13140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.803786 13126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:44:47.803962 13126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "8f26c629715947beadbb28dcfc6a3a40"
format_stamp: "Formatted at 2026-06-19 02:44:47 on dist-test-slave-9tkz"
server_key: "1bc24a21c16b3419d81c7b46d844e3f3"
server_key_iv: "5daca7313e039e64f10fb12951f82d93"
server_key_version: "encryptionkey@0"
I20260619 02:44:47.804185 13126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:47.809896 13126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:47.810731 13126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:47.811028 13126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:47.811946 13126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:47.813331 13126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:47.813452 13126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.813632 13126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:47.813724 13126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:47.846952 13126 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:42561
I20260619 02:44:47.847007 13252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:42561 every 8 connection(s)
I20260619 02:44:47.848346 13126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:47.849440 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13126
I20260619 02:44:47.849642 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:47.850219 8153 external_mini_cluster.cc:1428] Setting key 31e8600beb411e33f236516cf26ec9d9
I20260619 02:44:47.856993 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.851351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:46571 (local address 127.7.246.126:43965)
0619 02:44:47.851550 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.851567 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.852984 (+ 1417us) server_negotiation.cc:407] Connection header received
0619 02:44:47.854796 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.854801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.854862 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.854976 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.856251 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.856257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.856261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.856279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.856316 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.856327 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.856439 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.856657 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.856719 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:44:47.857985 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.851874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43965 (local address 127.7.246.67:46571)
0619 02:44:47.852819 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:47.852864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:47.854521 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:47.855223 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:47.855232 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:47.855336 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:47.855974 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:47.855977 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:47.856012 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:47.856093 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:47.856426 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:47.856436 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:47.856444 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:44:47.856741 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 02:44:47.857002 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
I20260619 02:44:47.859225 13253 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43965
I20260619 02:44:47.859472 13253 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:47.860028 13253 heartbeater.cc:511] Master 127.7.246.126:43965 requested a full tablet report, sending...
I20260619 02:44:47.861332 12806 ts_manager.cc:194] Registered new tserver with Master: 8f26c629715947beadbb28dcfc6a3a40 (127.7.246.67:42561)
I20260619 02:44:47.865425 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:47.872622 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.869943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.7.246.126:43965)
0619 02:44:47.870140 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.870149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.870732 (+ 583us) server_negotiation.cc:407] Connection header received
0619 02:44:47.870843 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.870848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.870916 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.871011 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.871964 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.871969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.871973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.871989 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.872023 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.872035 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.872145 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.872225 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.872271 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:44:47.877162 12806 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:44:47.884977 12806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:44:47.914974 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.898316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56856)
0619 02:44:47.910470 (+ 12154us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:47.910489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:47.910645 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:47.912499 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:47.912509 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:47.912614 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:47.913446 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:47.913451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:47.913486 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:47.913617 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:47.914172 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:47.914184 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:47.914193 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:44:47.914384 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 02:44:47.914481 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12020,"thread_start_us":11903,"threads_started":1}
I20260619 02:44:47.915932 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.898573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.7.246.65:43449)
0619 02:44:47.909012 (+ 10439us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.909024 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.910785 (+ 1761us) server_negotiation.cc:407] Connection header received
0619 02:44:47.910868 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.910877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.911194 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.911374 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.913821 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.913832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.913850 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.913873 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.913937 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.913959 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.914116 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.915490 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.915596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10266,"thread_start_us":10135,"threads_started":1}
I20260619 02:44:47.919639 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.897853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37237 (local address 127.0.0.1:53330)
0619 02:44:47.909640 (+ 11787us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:47.909706 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:47.909912 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:47.916972 (+ 7060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:47.916979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:47.917024 (+ 45us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:47.917724 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:47.917729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:47.917746 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:47.917795 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:47.918996 (+ 1201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:47.919003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:47.919012 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:44:47.919143 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 02:44:47.919200 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11423,"thread_start_us":11211,"threads_started":1}
I20260619 02:44:47.920357 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.913620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:42561 (local address 127.0.0.1:49322)
0619 02:44:47.915232 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:47.915251 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:47.915329 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:47.917417 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:47.917421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:47.917465 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:47.917993 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:47.917997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:47.918016 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:47.918080 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:47.919919 (+ 1839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:47.919925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:47.919930 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:44:47.920062 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:44:47.920111 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"thread_start_us":4623,"threads_started":1}
I20260619 02:44:47.920676 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.897997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.7.246.66:37237)
0619 02:44:47.916384 (+ 18387us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.916395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.916430 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:44:47.916500 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.916507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.916764 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.917421 (+ 657us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.918730 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.918739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.918756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.918775 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.918838 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.918855 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.920155 (+ 1300us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.920234 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.920337 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18194,"thread_start_us":15158,"threads_started":1}
I20260619 02:44:47.920895 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:47.910730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.7.246.67:42561)
0619 02:44:47.916953 (+ 6223us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:47.916961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:47.917012 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:44:47.917088 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:47.917095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:47.917249 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:47.917439 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:47.919581 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:47.919586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:47.919596 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:47.919608 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:47.919642 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:47.919654 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:47.919853 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:47.920087 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 02:44:47.920178 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5986,"thread_start_us":6069,"threads_started":1}
I20260619 02:44:47.924533 12927 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc909fee7640fea1a73479276c0825 (DEFAULT_TABLE table=test-table [id=32a4303fcfc84d7a94dedd02dcbdbebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:47.924533 13058 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc909fee7640fea1a73479276c0825 (DEFAULT_TABLE table=test-table [id=32a4303fcfc84d7a94dedd02dcbdbebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:47.924816 13188 tablet_service.cc:1467] Processing CreateTablet for tablet 13bc909fee7640fea1a73479276c0825 (DEFAULT_TABLE table=test-table [id=32a4303fcfc84d7a94dedd02dcbdbebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:44:47.926002 13188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc909fee7640fea1a73479276c0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:47.926002 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc909fee7640fea1a73479276c0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:47.926010 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13bc909fee7640fea1a73479276c0825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:47.941306 13271 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Bootstrap starting.
I20260619 02:44:47.946182 13271 tablet_bootstrap.cc:654] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:47.947715 13271 log.cc:824] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:47.947804 13272 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Bootstrap starting.
I20260619 02:44:47.953305 13273 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Bootstrap starting.
I20260619 02:44:47.953845 13272 tablet_bootstrap.cc:654] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:47.955103 13272 log.cc:824] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:47.955677 13271 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: No bootstrap required, opened a new log
I20260619 02:44:47.956041 13271 ts_tablet_manager.cc:1397] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260619 02:44:47.966355 13273 tablet_bootstrap.cc:654] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:47.970944 13273 log.cc:824] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:47.974718 13272 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: No bootstrap required, opened a new log
I20260619 02:44:47.974460 13271 raft_consensus.cc:348] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.974959 13271 raft_consensus.cc:374] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:47.975015 13272 ts_tablet_manager.cc:1397] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.011s
I20260619 02:44:47.975073 13271 raft_consensus.cc:729] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc591455aae4e018ee55d7cc99459c6, State: Initialized, Role: FOLLOWER
I20260619 02:44:47.975342 13273 tablet_bootstrap.cc:492] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: No bootstrap required, opened a new log
I20260619 02:44:47.975670 13273 ts_tablet_manager.cc:1397] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260619 02:44:47.975898 13271 consensus_queue.cc:260] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.984819 13271 ts_tablet_manager.cc:1428] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Time spent starting tablet: real 0.029s user 0.015s sys 0.006s
I20260619 02:44:47.985185 12992 heartbeater.cc:503] Master 127.7.246.126:43965 was elected leader, sending a full tablet report...
I20260619 02:44:47.987807 13272 raft_consensus.cc:348] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.988313 13272 raft_consensus.cc:374] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:47.988420 13272 raft_consensus.cc:729] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5a9bf52f0c43fe88d91323bd0bb499, State: Initialized, Role: FOLLOWER
I20260619 02:44:47.989229 13272 consensus_queue.cc:260] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.989009 13273 raft_consensus.cc:348] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.990171 13273 raft_consensus.cc:374] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:47.990301 13273 raft_consensus.cc:729] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f26c629715947beadbb28dcfc6a3a40, State: Initialized, Role: FOLLOWER
I20260619 02:44:47.991091 13273 consensus_queue.cc:260] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:47.992172 13123 heartbeater.cc:503] Master 127.7.246.126:43965 was elected leader, sending a full tablet report...
I20260619 02:44:47.992919 13272 ts_tablet_manager.cc:1428] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260619 02:44:47.993790 13253 heartbeater.cc:503] Master 127.7.246.126:43965 was elected leader, sending a full tablet report...
I20260619 02:44:47.994542 13273 ts_tablet_manager.cc:1428] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Time spent starting tablet: real 0.019s user 0.010s sys 0.005s
I20260619 02:44:48.014534 13279 raft_consensus.cc:482] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:44:48.014853 13279 raft_consensus.cc:504] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:48.016539 13279 leader_election.cc:288] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce5a9bf52f0c43fe88d91323bd0bb499 (127.7.246.66:37237), adc591455aae4e018ee55d7cc99459c6 (127.7.246.65:43449)
I20260619 02:44:48.019565 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.017051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:41907 (local address 127.7.246.65:43449)
0619 02:44:48.017265 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.017274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.017302 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:44:48.017595 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.017600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.017663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.017780 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.018894 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.018901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.018907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.018926 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.018963 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.018982 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.019111 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.019226 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.019272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:44:48.019588 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.016912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.7.246.67:41907)
0619 02:44:48.017200 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.017222 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:48.017356 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:48.017844 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:48.017850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:48.017878 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:48.018624 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:48.018630 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:48.018654 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:48.018714 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:48.019085 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:48.019094 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:48.019102 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:44:48.019237 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:44:48.019296 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 02:44:48.020077 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.016919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:54005 (local address 127.7.246.66:37237)
0619 02:44:48.017113 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.017119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.017973 (+ 854us) server_negotiation.cc:407] Connection header received
0619 02:44:48.018161 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.018165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.018253 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.018353 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.019378 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.019383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.019389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.019407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.019441 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.019455 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.019569 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.019758 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.019824 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:44:48.020567 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc909fee7640fea1a73479276c0825" candidate_uuid: "8f26c629715947beadbb28dcfc6a3a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc591455aae4e018ee55d7cc99459c6" is_pre_election: true
I20260619 02:44:48.020761 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.016996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37237 (local address 127.7.246.67:54005)
0619 02:44:48.017844 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.017868 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:48.018006 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:48.018401 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:48.018407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:48.018455 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:48.019171 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:44:48.019176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:48.019197 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:44:48.019239 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:48.019563 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:48.019571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:48.019583 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:44:48.020327 (+ 744us) client_negotiation.cc:240] Negotiation successful
0619 02:44:48.020386 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":482,"threads_started":1}
I20260619 02:44:48.021087 12947 raft_consensus.cc:2454] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f26c629715947beadbb28dcfc6a3a40 in term 0.
I20260619 02:44:48.021818 13143 leader_election.cc:302] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f26c629715947beadbb28dcfc6a3a40, adc591455aae4e018ee55d7cc99459c6; no voters:
I20260619 02:44:48.021819 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc909fee7640fea1a73479276c0825" candidate_uuid: "8f26c629715947beadbb28dcfc6a3a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" is_pre_election: true
I20260619 02:44:48.022310 13279 raft_consensus.cc:2781] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:44:48.022401 13078 raft_consensus.cc:2454] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f26c629715947beadbb28dcfc6a3a40 in term 0.
I20260619 02:44:48.022488 13279 raft_consensus.cc:482] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:44:48.022598 13279 raft_consensus.cc:3037] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:48.025091 13279 raft_consensus.cc:504] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:48.025830 13279 leader_election.cc:288] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [CANDIDATE]: Term 1 election: Requested vote from peers ce5a9bf52f0c43fe88d91323bd0bb499 (127.7.246.66:37237), adc591455aae4e018ee55d7cc99459c6 (127.7.246.65:43449)
I20260619 02:44:48.026311 13078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc909fee7640fea1a73479276c0825" candidate_uuid: "8f26c629715947beadbb28dcfc6a3a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499"
I20260619 02:44:48.026415 12947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13bc909fee7640fea1a73479276c0825" candidate_uuid: "8f26c629715947beadbb28dcfc6a3a40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc591455aae4e018ee55d7cc99459c6"
I20260619 02:44:48.026535 13078 raft_consensus.cc:3037] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:48.026697 12947 raft_consensus.cc:3037] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:48.028867 13078 raft_consensus.cc:2454] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f26c629715947beadbb28dcfc6a3a40 in term 1.
I20260619 02:44:48.029093 12947 raft_consensus.cc:2454] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f26c629715947beadbb28dcfc6a3a40 in term 1.
I20260619 02:44:48.029327 13142 leader_election.cc:302] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f26c629715947beadbb28dcfc6a3a40, ce5a9bf52f0c43fe88d91323bd0bb499; no voters:
I20260619 02:44:48.029740 13279 raft_consensus.cc:2781] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:48.030223 13279 raft_consensus.cc:686] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 1 LEADER]: Becoming Leader. State: Replica: 8f26c629715947beadbb28dcfc6a3a40, State: Running, Role: LEADER
I20260619 02:44:48.030766 13279 consensus_queue.cc:237] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } }
I20260619 02:44:48.035542 12806 catalog_manager.cc:5495] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f26c629715947beadbb28dcfc6a3a40 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "8f26c629715947beadbb28dcfc6a3a40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f26c629715947beadbb28dcfc6a3a40" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 42561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 } health_report { overall_health: UNKNOWN } } }
I20260619 02:44:48.060153 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.057929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.7.246.67:42561)
0619 02:44:48.058121 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.058128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.058253 (+ 125us) server_negotiation.cc:407] Connection header received
0619 02:44:48.058508 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.058512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.058591 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.058705 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.059564 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.059569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.059572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.059586 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.059615 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.059627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.059756 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.059842 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.059900 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:44:48.068518 13078 raft_consensus.cc:1261] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 1 FOLLOWER]: Refusing update from remote peer 8f26c629715947beadbb28dcfc6a3a40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:48.068517 12947 raft_consensus.cc:1261] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 1 FOLLOWER]: Refusing update from remote peer 8f26c629715947beadbb28dcfc6a3a40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:44:48.069509 13279 consensus_queue.cc:1035] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5a9bf52f0c43fe88d91323bd0bb499" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:48.069850 13281 consensus_queue.cc:1035] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc591455aae4e018ee55d7cc99459c6" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:44:48.084599 13285 mvcc.cc:205] Tried to move back new op lower bound from 7298404712718127104 to 7298404712578568192. Current Snapshot: MvccSnapshot[applied={T|T < 7298404712718127104 or (T in {7298404712718127104})}]
I20260619 02:44:48.099637 12806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:48.099936 12806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:44:48.103663 12806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51ed5ab583d94d30aefc0e1cdddbfce6: Sending DeleteTablet for 3 replicas of tablet 13bc909fee7640fea1a73479276c0825
I20260619 02:44:48.104969 13058 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc909fee7640fea1a73479276c0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:48 UTC) from {username='slave'} at 127.0.0.1:53330
I20260619 02:44:48.104974 13188 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc909fee7640fea1a73479276c0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:48 UTC) from {username='slave'} at 127.0.0.1:49322
I20260619 02:44:48.105031 12927 tablet_service.cc:1514] Processing DeleteTablet for tablet 13bc909fee7640fea1a73479276c0825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:44:48 UTC) from {username='slave'} at 127.0.0.1:56856
I20260619 02:44:48.107581 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12865
I20260619 02:44:48.118479 13294 tablet_replica.cc:330] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: stopping tablet replica
I20260619 02:44:48.119313 13294 raft_consensus.cc:2229] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:48.120110 13294 raft_consensus.cc:2258] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:48.121066 13295 tablet_replica.cc:330] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: stopping tablet replica
I20260619 02:44:48.121466 13295 raft_consensus.cc:2229] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:44:48.122874 13296 tablet_replica.cc:330] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: stopping tablet replica
I20260619 02:44:48.123157 13295 raft_consensus.cc:2258] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:48.123698 13296 raft_consensus.cc:2229] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:44:48.124259 13296 raft_consensus.cc:2258] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:44:48.124286 13294 ts_tablet_manager.cc:1905] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:48.124645 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.109011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.7.246.65:43449)
0619 02:44:48.109225 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.109234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.122182 (+ 12948us) server_negotiation.cc:407] Connection header received
0619 02:44:48.122406 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.122414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.122546 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.122685 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.123829 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.123835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.123840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.123860 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.123898 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.123914 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.124018 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.124260 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.124322 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:44:48.125768 12987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:48.129773 13294 ts_tablet_manager.cc:1918] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:48.129961 13294 log.cc:1196] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/13bc909fee7640fea1a73479276c0825
I20260619 02:44:48.130493 13294 ts_tablet_manager.cc:1939] T 13bc909fee7640fea1a73479276c0825 P ce5a9bf52f0c43fe88d91323bd0bb499: Deleting consensus metadata
I20260619 02:44:48.131819 13295 ts_tablet_manager.cc:1905] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:48.132342 12791 catalog_manager.cc:4841] TS ce5a9bf52f0c43fe88d91323bd0bb499 (127.7.246.66:37237): tablet 13bc909fee7640fea1a73479276c0825 (table test-table [id=32a4303fcfc84d7a94dedd02dcbdbebb]) successfully deleted
I20260619 02:44:48.139096 13295 ts_tablet_manager.cc:1918] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:44:48.139289 13295 log.cc:1196] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/13bc909fee7640fea1a73479276c0825
I20260619 02:44:48.139797 13295 ts_tablet_manager.cc:1939] T 13bc909fee7640fea1a73479276c0825 P 8f26c629715947beadbb28dcfc6a3a40: Deleting consensus metadata
I20260619 02:44:48.141311 12791 catalog_manager.cc:4841] TS 8f26c629715947beadbb28dcfc6a3a40 (127.7.246.67:42561): tablet 13bc909fee7640fea1a73479276c0825 (table test-table [id=32a4303fcfc84d7a94dedd02dcbdbebb]) successfully deleted
I20260619 02:44:48.126685 13296 ts_tablet_manager.cc:1905] T 13bc909fee7640fea1a73479276c0825 P adc591455aae4e018ee55d7cc99459c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:44:48.262246 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12865
I20260619 02:44:48.280004 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12996
W20260619 02:44:48.279987 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:43449 (error 108)
I20260619 02:44:48.282876 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.282373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56864)
0619 02:44:48.282547 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.282688 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:44:48.283272 12792 catalog_manager.cc:4584] TS adc591455aae4e018ee55d7cc99459c6 (127.7.246.65:43449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13bc909fee7640fea1a73479276c0825: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
I20260619 02:44:48.284588 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.281152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.7.246.66:37237)
0619 02:44:48.281496 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.281508 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.281534 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:44:48.281666 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.281675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.281801 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.281947 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.283603 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.283611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.283619 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.283643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.283687 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.283701 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.283818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.284203 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.284267 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:44:48.285434 13118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:48.337213 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.336663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56874)
0619 02:44:48.336922 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.337030 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 02:44:48.338510 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.338046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56888)
0619 02:44:48.338232 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.338334 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:44:48.373049 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12996
I20260619 02:44:48.394657 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13126
I20260619 02:44:48.397872 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.395701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.7.246.67:42561)
0619 02:44:48.395908 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:48.395917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:48.395941 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:44:48.396033 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:48.396037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:48.396084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:48.396204 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:48.397129 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:48.397134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:48.397141 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:48.397160 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:48.397195 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:48.397217 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:48.397370 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:48.397465 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:44:48.397536 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:44:48.398813 13248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:48.399106 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.398646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56904)
0619 02:44:48.398813 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.398929 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 02:44:48.400173 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.399796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:56916)
0619 02:44:48.399961 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.400051 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:44:48.480635 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.480165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:45520)
0619 02:44:48.480401 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.480483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 02:44:48.481807 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:48.481376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43449 (local address 127.0.0.1:45526)
0619 02:44:48.481637 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:48.481703 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:44:48.529150 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13126
I20260619 02:44:48.547590 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 12775
I20260619 02:44:48.548695 12836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:48.656394 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 12775
2026-06-19T02:44:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2717 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:44:48.697655 8153 mini_oidc.cc:152] Starting JWKS server
I20260619 02:44:48.700234 8153 webserver.cc:460] Webserver started at https://127.0.0.1:34957/ using document root <none> and password file <none>
I20260619 02:44:48.700543 8153 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:44:48.701189 8153 webserver.cc:460] Webserver started at http://127.0.0.1:44767/ using document root <none> and password file <none>
2026-06-19T02:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:48Z Disabled control of system clock
I20260619 02:44:48.727586 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:46127
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:46127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34957/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/testchainca.cert with env {}
W20260619 02:44:49.052485 13308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:49.052776 13308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:49.052862 13308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:49.060617 13308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:49.060731 13308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:49.060768 13308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:49.060817 13308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:49.060863 13308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:44:49.060935 13308 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:44:49.060976 13308 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34957/jwks.json
I20260619 02:44:49.071072 13308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:46127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34957/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:49.072314 13308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:49.073532 13308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:49.088919 13313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:49.088950 13314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:49.089073 13316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:49.090405 13308 server_base.cc:1034] running on GCE node
I20260619 02:44:49.091434 13308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:49.093801 13308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:49.095073 13308 hybrid_clock.cc:648] HybridClock initialized: now 1781837089094993 us; error 57 us; skew 500 ppm
I20260619 02:44:49.095691 13308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:49.098692 13308 webserver.cc:460] Webserver started at http://127.7.246.126:33761/ using document root <none> and password file <none>
I20260619 02:44:49.099431 13308 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:49.099527 13308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:49.099790 13308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:49.103281 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "66f6518caa514ed89085dd2aa3730d4e"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1169a4c36964016c4ca5823c70c9122"
server_key_iv: "c2992e796913f31cfd829b568c8cacb0"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.104377 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "66f6518caa514ed89085dd2aa3730d4e"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1169a4c36964016c4ca5823c70c9122"
server_key_iv: "c2992e796913f31cfd829b568c8cacb0"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.111702 13308 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260619 02:44:49.116235 13322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:49.117935 13308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:44:49.118144 13308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "66f6518caa514ed89085dd2aa3730d4e"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1169a4c36964016c4ca5823c70c9122"
server_key_iv: "c2992e796913f31cfd829b568c8cacb0"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.118371 13308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:49.141038 13308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:49.142092 13308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:49.142418 13308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:49.161118 13308 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:46127
I20260619 02:44:49.161142 13373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:46127 every 8 connection(s)
I20260619 02:44:49.162545 13308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:49.166527 13374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:49.166947 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13308
I20260619 02:44:49.167132 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:49.167749 8153 external_mini_cluster.cc:1428] Setting key db3cb0661cbc6a3ceee07209ed26bb08
I20260619 02:44:49.180401 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e: Bootstrap starting.
I20260619 02:44:49.182411 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:49.169699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.7.246.126:46127)
0619 02:44:49.170877 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:49.170889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:49.170932 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:44:49.172591 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:49.172616 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:49.173145 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:49.173505 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:49.174198 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.175438 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.177407 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.178276 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.178408 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:49.179742 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:49.179752 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:49.179768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:49.180352 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:49.180454 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:49.180478 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:49.180654 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:49.180903 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:44:49.181259 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":357,"threads_started":1}
I20260619 02:44:49.183946 13374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:49.185103 13374 log.cc:824] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:49.186687 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:44:49.188578 13374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e: No bootstrap required, opened a new log
I20260619 02:44:49.196950 13374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } }
I20260619 02:44:49.197332 13374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:49.197417 13374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f6518caa514ed89085dd2aa3730d4e, State: Initialized, Role: FOLLOWER
I20260619 02:44:49.198181 13374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } }
I20260619 02:44:49.198505 13374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:49.198676 13374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:49.198866 13374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:49.201157 13374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } }
I20260619 02:44:49.201798 13374 leader_election.cc:302] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66f6518caa514ed89085dd2aa3730d4e; no voters:
I20260619 02:44:49.202406 13374 leader_election.cc:288] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:49.202551 13379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:49.204818 13374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:49.205260 13379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [term 1 LEADER]: Becoming Leader. State: Replica: 66f6518caa514ed89085dd2aa3730d4e, State: Running, Role: LEADER
I20260619 02:44:49.205834 13379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } }
I20260619 02:44:49.208971 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66f6518caa514ed89085dd2aa3730d4e. Latest consensus state: current_term: 1 leader_uuid: "66f6518caa514ed89085dd2aa3730d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } } }
I20260619 02:44:49.209414 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:49.209085 13380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66f6518caa514ed89085dd2aa3730d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66f6518caa514ed89085dd2aa3730d4e" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 46127 } } }
I20260619 02:44:49.209582 13380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:49.211226 13389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:49.216428 13389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:49.225126 13389 catalog_manager.cc:1347] Generated new cluster ID: 328aec601ed8449fa4093385d497c035
I20260619 02:44:49.225248 13389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:49.247118 13389 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:49.248414 13389 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:49.258976 13389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66f6518caa514ed89085dd2aa3730d4e: Generated new TSK 0
I20260619 02:44:49.259817 13389 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:44:49.301632 13404 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:44:49.301836 13404 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:44:49.302989 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:49.280835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.7.246.126:46127)
0619 02:44:49.281036 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:49.281044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:49.281610 (+ 566us) server_negotiation.cc:407] Connection header received
0619 02:44:49.282009 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:49.282016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:49.282106 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:49.282274 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:44:49.282780 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.283943 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.285461 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.286201 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.286331 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:49.286861 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:44:49.302323 (+ 15462us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:44:49.302688 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 02:44:49.303113 13377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:44:49.304312 13403 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:49.280677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:46127 (local address 127.0.0.1:46238)
0619 02:44:49.281476 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:49.281492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:49.281757 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:49.282299 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:49.282302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:49.282312 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:44:49.282625 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:49.282633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.284166 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.284170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:49.285309 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:49.285319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.286376 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.286380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:49.286508 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:49.286568 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:44:49.302603 (+ 16035us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:44:49.302871 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.246.126:46127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":461,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.7.246.126:46127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34957/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:44:49.306046 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13308
I20260619 02:44:49.307256 13369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:49.401074 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13308
2026-06-19T02:44:49Z chronyd exiting
I20260619 02:44:49.451404 8153 test_util.cc:175] -----------------------------------------------
I20260619 02:44:49.451535 8153 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837047186258-8153-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:44:49.455253 8153 mini_oidc.cc:152] Starting JWKS server
I20260619 02:44:49.457134 8153 webserver.cc:460] Webserver started at https://127.0.0.1:42051/ using document root <none> and password file <none>
I20260619 02:44:49.457387 8153 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:44:49.458053 8153 webserver.cc:460] Webserver started at http://127.0.0.1:45127/ using document root <none> and password file <none>
2026-06-19T02:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:49Z Disabled control of system clock
I20260619 02:44:49.484217 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38029
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:38029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42051/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/testchainca.cert with env {}
W20260619 02:44:49.807832 13411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:49.808176 13411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:49.808328 13411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:49.815600 13411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:49.815758 13411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:49.815852 13411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:49.815966 13411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:49.816051 13411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:44:49.816140 13411 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:44:49.816216 13411 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42051/jwks.json
I20260619 02:44:49.826287 13411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:38029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42051/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:49.827641 13411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:49.828975 13411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:49.840873 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:49.840991 13417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:49.841241 13416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:49.842219 13411 server_base.cc:1034] running on GCE node
I20260619 02:44:49.843037 13411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:49.844463 13411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:49.845750 13411 hybrid_clock.cc:648] HybridClock initialized: now 1781837089845694 us; error 53 us; skew 500 ppm
I20260619 02:44:49.846093 13411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:49.848229 13411 webserver.cc:460] Webserver started at http://127.7.246.126:33955/ using document root <none> and password file <none>
I20260619 02:44:49.848907 13411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:49.849071 13411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:49.849386 13411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:49.852262 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "a3b2ca403f6541bf9a856a404bce1b81"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1566f0f50627e5cb3d7c15c2ca88eec"
server_key_iv: "7bdf6e564b4b429bcf7f569adf8fc87e"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.853041 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "a3b2ca403f6541bf9a856a404bce1b81"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1566f0f50627e5cb3d7c15c2ca88eec"
server_key_iv: "7bdf6e564b4b429bcf7f569adf8fc87e"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.858064 13411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:44:49.861498 13425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:49.862922 13411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:49.863109 13411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "a3b2ca403f6541bf9a856a404bce1b81"
format_stamp: "Formatted at 2026-06-19 02:44:49 on dist-test-slave-9tkz"
server_key: "f1566f0f50627e5cb3d7c15c2ca88eec"
server_key_iv: "7bdf6e564b4b429bcf7f569adf8fc87e"
server_key_version: "encryptionkey@0"
I20260619 02:44:49.863320 13411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:49.876569 13411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:49.877382 13411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:49.877743 13411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:49.896412 13411 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:38029
I20260619 02:44:49.896436 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:38029 every 8 connection(s)
I20260619 02:44:49.897845 13411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:49.901621 13477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:49.903360 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13411
I20260619 02:44:49.903589 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:49.904374 8153 external_mini_cluster.cc:1428] Setting key db7c45257a48547699fdeb760682a4c6
I20260619 02:44:49.914712 13477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81: Bootstrap starting.
I20260619 02:44:49.917141 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:49.906319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.7.246.126:38029)
0619 02:44:49.907414 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:49.907425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:49.907464 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:44:49.908974 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:49.908994 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:49.909446 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:49.909868 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:49.910384 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.911610 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.912858 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.913511 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.913775 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:49.914969 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:49.914976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:49.914988 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:49.915486 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:49.915554 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:49.915573 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:49.915700 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:49.915911 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:44:49.916238 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":432,"threads_started":1}
I20260619 02:44:49.918088 13477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:49.919276 13477 log.cc:824] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:49.920693 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:44:49.922144 13477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81: No bootstrap required, opened a new log
I20260619 02:44:49.929172 13477 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } }
I20260619 02:44:49.929601 13477 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:49.929749 13477 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b2ca403f6541bf9a856a404bce1b81, State: Initialized, Role: FOLLOWER
I20260619 02:44:49.930377 13477 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } }
I20260619 02:44:49.930583 13477 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:49.930675 13477 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:49.930789 13477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:49.933022 13477 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } }
I20260619 02:44:49.933461 13477 leader_election.cc:302] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b2ca403f6541bf9a856a404bce1b81; no voters:
I20260619 02:44:49.934005 13477 leader_election.cc:288] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:49.934154 13482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:49.934861 13482 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [term 1 LEADER]: Becoming Leader. State: Replica: a3b2ca403f6541bf9a856a404bce1b81, State: Running, Role: LEADER
I20260619 02:44:49.935420 13482 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } }
I20260619 02:44:49.936082 13477 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:49.938264 13483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b2ca403f6541bf9a856a404bce1b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } } }
I20260619 02:44:49.938587 13483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:49.939270 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b2ca403f6541bf9a856a404bce1b81. Latest consensus state: current_term: 1 leader_uuid: "a3b2ca403f6541bf9a856a404bce1b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b2ca403f6541bf9a856a404bce1b81" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38029 } } }
I20260619 02:44:49.939550 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:49.940291 13490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:49.945518 13490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:49.954329 13490 catalog_manager.cc:1347] Generated new cluster ID: c5c15eee43344003b41154fbfd7583b2
I20260619 02:44:49.954463 13490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:49.968606 13490 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:49.970031 13490 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:49.980237 13490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3b2ca403f6541bf9a856a404bce1b81: Generated new TSK 0
I20260619 02:44:49.981331 13490 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:44:50.015349 13507 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:44:50.015486 13507 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:44:50.016541 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:49.997021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.7.246.126:38029)
0619 02:44:49.997236 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:49.997246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:49.997707 (+ 461us) server_negotiation.cc:407] Connection header received
0619 02:44:49.997951 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:49.997955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:49.998035 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:49.998178 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:44:49.998733 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.999443 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.001385 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.001907 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.002003 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:50.002585 (+ 582us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:44:50.016018 (+ 13433us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:44:50.016313 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 02:44:50.016696 13480 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 02:44:50.016985 13506 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:49.996861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:38029 (local address 127.0.0.1:58744)
0619 02:44:49.997600 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:49.997621 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:49.997780 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:49.998224 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:49.998229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:49.998242 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:44:49.998562 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:49.998574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:49.999588 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:49.999594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:50.001249 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:50.001263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.002086 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.002093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:50.002278 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:50.002324 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:44:50.016272 (+ 13948us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:44:50.016542 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.246.126:38029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":390,"threads_started":1}
I20260619 02:44:50.018256 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13411
I20260619 02:44:50.019505 13472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:50.114229 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13411
2026-06-19T02:44:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:44:50.169016 8153 mini_oidc.cc:152] Starting JWKS server
I20260619 02:44:50.170924 8153 webserver.cc:460] Webserver started at https://127.0.0.1:43083/ using document root <none> and password file <none>
I20260619 02:44:50.171176 8153 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:44:50.171792 8153 webserver.cc:460] Webserver started at http://127.0.0.1:45251/ using document root <none> and password file <none>
2026-06-19T02:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:50Z Disabled control of system clock
I20260619 02:44:50.199172 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34923
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34923
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43083/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:44:50.517992 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:50.518392 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:50.518563 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:50.525925 13514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:50.526105 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:50.526201 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:50.526304 13514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:50.526391 13514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:44:50.526480 13514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:44:50.526538 13514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43083/jwks.json
I20260619 02:44:50.535704 13514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43083/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:50.537175 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:50.539065 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:50.550963 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:50.551012 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:50.551028 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:50.552311 13514 server_base.cc:1034] running on GCE node
I20260619 02:44:50.553063 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:50.554616 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:50.555871 13514 hybrid_clock.cc:648] HybridClock initialized: now 1781837090555825 us; error 72 us; skew 500 ppm
I20260619 02:44:50.556260 13514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:50.558792 13514 webserver.cc:460] Webserver started at http://127.7.246.126:39695/ using document root <none> and password file <none>
I20260619 02:44:50.559511 13514 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:50.559643 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:50.559969 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:50.562987 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "249519518d62431ca430c68e422c9e52"
format_stamp: "Formatted at 2026-06-19 02:44:50 on dist-test-slave-9tkz"
server_key: "6ab294300d480ebdaf02327043f7d540"
server_key_iv: "7840f56b96aa290d37a5b5d8111cebd4"
server_key_version: "encryptionkey@0"
I20260619 02:44:50.563794 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "249519518d62431ca430c68e422c9e52"
format_stamp: "Formatted at 2026-06-19 02:44:50 on dist-test-slave-9tkz"
server_key: "6ab294300d480ebdaf02327043f7d540"
server_key_iv: "7840f56b96aa290d37a5b5d8111cebd4"
server_key_version: "encryptionkey@0"
I20260619 02:44:50.568904 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 02:44:50.572543 13528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:50.573926 13514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:44:50.574115 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "249519518d62431ca430c68e422c9e52"
format_stamp: "Formatted at 2026-06-19 02:44:50 on dist-test-slave-9tkz"
server_key: "6ab294300d480ebdaf02327043f7d540"
server_key_iv: "7840f56b96aa290d37a5b5d8111cebd4"
server_key_version: "encryptionkey@0"
I20260619 02:44:50.574344 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:50.594695 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:50.595598 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:50.595916 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:50.615025 13514 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34923
I20260619 02:44:50.615056 13579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34923 every 8 connection(s)
I20260619 02:44:50.616436 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:50.618544 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13514
I20260619 02:44:50.618743 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:50.619378 8153 external_mini_cluster.cc:1428] Setting key 4098be1a276224978528185a69ddff6a
I20260619 02:44:50.622038 13580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:50.632442 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:50.621143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.7.246.126:34923)
0619 02:44:50.622233 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:50.622247 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:50.622292 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:44:50.623828 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:50.623853 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:50.624327 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:50.624752 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:50.625316 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.626681 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.628003 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.628678 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.628884 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:50.630062 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:50.630069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:50.630093 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:50.630637 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:50.630715 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:50.630734 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:50.630870 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:50.631112 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:44:50.631440 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260619 02:44:50.634231 13580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52: Bootstrap starting.
I20260619 02:44:50.636536 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:44:50.638020 13580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:50.639282 13580 log.cc:824] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:50.642462 13580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52: No bootstrap required, opened a new log
I20260619 02:44:50.651093 13580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } }
I20260619 02:44:50.651559 13580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:50.651683 13580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249519518d62431ca430c68e422c9e52, State: Initialized, Role: FOLLOWER
I20260619 02:44:50.652295 13580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } }
I20260619 02:44:50.652565 13580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:50.652719 13580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:50.652913 13580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:50.655616 13580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } }
I20260619 02:44:50.656177 13580 leader_election.cc:302] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249519518d62431ca430c68e422c9e52; no voters:
I20260619 02:44:50.656782 13580 leader_election.cc:288] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:50.656885 13585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:50.657765 13585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [term 1 LEADER]: Becoming Leader. State: Replica: 249519518d62431ca430c68e422c9e52, State: Running, Role: LEADER
I20260619 02:44:50.658362 13585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } }
I20260619 02:44:50.659296 13580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:50.661562 13586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249519518d62431ca430c68e422c9e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } } }
I20260619 02:44:50.662106 13586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:50.661708 13587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249519518d62431ca430c68e422c9e52. Latest consensus state: current_term: 1 leader_uuid: "249519518d62431ca430c68e422c9e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249519518d62431ca430c68e422c9e52" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34923 } } }
I20260619 02:44:50.664508 13590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:50.664565 13587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:50.670760 13590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:50.688336 13590 catalog_manager.cc:1347] Generated new cluster ID: 2c82b010af044abba0d7d1da2afb683b
I20260619 02:44:50.688481 13590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:50.717087 13590 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:50.718665 13590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:50.726238 13590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249519518d62431ca430c68e422c9e52: Generated new TSK 0
I20260619 02:44:50.727387 13590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:44:50.766708 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:50.757860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.7.246.126:34923)
0619 02:44:50.758086 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:50.758097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:50.758538 (+ 441us) server_negotiation.cc:407] Connection header received
0619 02:44:50.758759 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:50.758766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:50.758866 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:50.759044 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:44:50.759531 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.760666 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.762162 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.762897 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.763021 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:50.763530 (+ 509us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:44:50.766004 (+ 2474us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:44:50.766353 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 02:44:50.766839 13583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 02:44:50.768365 13609 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:50.757777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34923 (local address 127.0.0.1:34044)
0619 02:44:50.758432 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:50.758449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:50.758570 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:50.759068 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:50.759079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:50.759089 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:44:50.759379 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:50.759387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.760838 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.760842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:50.762015 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:50.762026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:50.763088 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:50.763092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:50.763210 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:50.763243 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:44:50.766243 (+ 3000us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:44:50.767924 (+ 1681us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.246.126:34923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43083/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":1367,"thread_start_us":384,"threads_started":1}
I20260619 02:44:50.769959 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13514
I20260619 02:44:50.771258 13575 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 02:44:50.780025 13610 webserver.cc:533] Webserver: sslize failed:
E20260619 02:44:50.780148 13610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:44:50.871786 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13514
2026-06-19T02:44:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:50Z Disabled control of system clock
I20260619 02:44:50.953442 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:33837
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:33837
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/testchainca.cert with env {}
W20260619 02:44:51.275344 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:51.275655 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:51.275810 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:51.283263 13615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:51.283380 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:51.283485 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:51.283601 13615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:51.283684 13615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:44:51.283774 13615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:44:51.283836 13615 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:44:51.294101 13615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:33837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:51.295464 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:51.296782 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:51.308682 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:51.308619 13623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:51.308660 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:51.309859 13615 server_base.cc:1034] running on GCE node
I20260619 02:44:51.310740 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:51.312296 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:51.313603 13615 hybrid_clock.cc:648] HybridClock initialized: now 1781837091313552 us; error 106 us; skew 500 ppm
I20260619 02:44:51.314005 13615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:44:51.316404 13615 webserver.cc:460] Webserver started at http://127.7.246.126:36069/ using document root <none> and password file <none>
I20260619 02:44:51.317117 13615 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:51.317255 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:51.317631 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:51.320549 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "bc44f7d6e16d428e9fe7def41a98ea64"
format_stamp: "Formatted at 2026-06-19 02:44:51 on dist-test-slave-9tkz"
server_key: "917c423f041b7df06b4925c908c879bd"
server_key_iv: "cc5f724a0d0d4cb188ec8ffa72a98ec2"
server_key_version: "encryptionkey@0"
I20260619 02:44:51.321348 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "bc44f7d6e16d428e9fe7def41a98ea64"
format_stamp: "Formatted at 2026-06-19 02:44:51 on dist-test-slave-9tkz"
server_key: "917c423f041b7df06b4925c908c879bd"
server_key_iv: "cc5f724a0d0d4cb188ec8ffa72a98ec2"
server_key_version: "encryptionkey@0"
I20260619 02:44:51.326743 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 02:44:51.330220 13629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:51.331557 13615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:44:51.331768 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "bc44f7d6e16d428e9fe7def41a98ea64"
format_stamp: "Formatted at 2026-06-19 02:44:51 on dist-test-slave-9tkz"
server_key: "917c423f041b7df06b4925c908c879bd"
server_key_iv: "cc5f724a0d0d4cb188ec8ffa72a98ec2"
server_key_version: "encryptionkey@0"
I20260619 02:44:51.331998 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:51.360664 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:51.361517 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:51.361805 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:51.380115 13615 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:33837
I20260619 02:44:51.380127 13680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:33837 every 8 connection(s)
I20260619 02:44:51.381467 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:51.383178 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13615
I20260619 02:44:51.383409 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:51.384001 8153 external_mini_cluster.cc:1428] Setting key bb5668152e3157da41630fe322e25397
I20260619 02:44:51.385723 13681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:51.399125 13681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64: Bootstrap starting.
I20260619 02:44:51.402362 13681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:51.403752 13681 log.cc:824] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:51.405113 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:51.385944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.7.246.126:33837)
0619 02:44:51.386977 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:51.386998 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:51.387047 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:44:51.388950 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:51.388978 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:51.389634 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:51.390093 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:51.390685 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.392695 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.394280 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.395320 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.395501 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:51.402270 (+ 6769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:51.402283 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:51.402303 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:44:51.403057 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:51.403170 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:44:51.403195 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:51.403321 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:51.403557 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 02:44:51.403901 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":309,"threads_started":1}
I20260619 02:44:51.406729 13681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64: No bootstrap required, opened a new log
I20260619 02:44:51.408815 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:44:51.414624 13681 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } }
I20260619 02:44:51.414963 13681 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:51.415067 13681 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc44f7d6e16d428e9fe7def41a98ea64, State: Initialized, Role: FOLLOWER
I20260619 02:44:51.415619 13681 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } }
I20260619 02:44:51.415843 13681 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:51.416009 13681 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:51.416183 13681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:51.418756 13681 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } }
I20260619 02:44:51.419257 13681 leader_election.cc:302] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc44f7d6e16d428e9fe7def41a98ea64; no voters:
I20260619 02:44:51.419842 13681 leader_election.cc:288] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:51.419946 13686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:51.420643 13686 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [term 1 LEADER]: Becoming Leader. State: Replica: bc44f7d6e16d428e9fe7def41a98ea64, State: Running, Role: LEADER
I20260619 02:44:51.421159 13686 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } }
I20260619 02:44:51.421988 13681 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:51.424031 13688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc44f7d6e16d428e9fe7def41a98ea64. Latest consensus state: current_term: 1 leader_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } } }
I20260619 02:44:51.423987 13687 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc44f7d6e16d428e9fe7def41a98ea64" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33837 } } }
I20260619 02:44:51.424469 13687 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:51.425256 13688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64 [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:51.425458 13694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:51.430611 13694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:51.439599 13694 catalog_manager.cc:1347] Generated new cluster ID: 2ad7a80b8f594b3299f86a8a246bae6f
I20260619 02:44:51.439781 13694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:51.460996 13694 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:51.462417 13694 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:51.471212 13694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc44f7d6e16d428e9fe7def41a98ea64: Generated new TSK 0
I20260619 02:44:51.472329 13694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:44:51.506539 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:51.487721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.7.246.126:33837)
0619 02:44:51.487918 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:51.487925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:51.488395 (+ 470us) server_negotiation.cc:407] Connection header received
0619 02:44:51.488629 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:51.488635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:51.488726 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:51.488895 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:44:51.489374 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.490588 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.492099 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.492817 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.492934 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:51.493406 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:44:51.505700 (+ 12294us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:44:51.506229 (+ 529us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46550: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 02:44:51.506673 13684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46550: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 02:44:51.506664 13710 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:51.487649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:33837 (local address 127.0.0.1:46550)
0619 02:44:51.488293 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:51.488308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:51.488449 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:51.488924 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:51.488928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:51.488938 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:44:51.489221 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:51.489229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.490801 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.490809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:51.491951 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:51.491962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:51.492983 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:51.492987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:51.493100 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:51.493132 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:44:51.506103 (+ 12971us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:44:51.506337 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.7.246.126:33837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":359,"threads_started":1}
I20260619 02:44:51.508550 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13615
I20260619 02:44:51.509754 13676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:51.602200 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13615
2026-06-19T02:44:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (709 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1485 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:53 dist-test-slave-9tkz krb5kdc[13726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:53 dist-test-slave-9tkz krb5kdc[13726](info): set up 2 sockets
Jun 19 02:44:53 dist-test-slave-9tkz krb5kdc[13726](info): commencing operation
krb5kdc: starting...
W20260619 02:44:54.562606 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:54 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:44:54.776847 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38967
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:38967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:55.104853 13739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:55.105149 13739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:55.105342 13739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:55.112650 13739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:55.112807 13739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:55.112918 13739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:55.113030 13739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:55.113122 13739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:55.123091 13739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:38967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:55.124433 13739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:55.125794 13739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:55.137528 13747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:55.137506 13745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:55.137671 13744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:55.138294 13739 server_base.cc:1034] running on GCE node
I20260619 02:44:55.138990 13739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:55.140232 13739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:55.141455 13739 hybrid_clock.cc:648] HybridClock initialized: now 1781837095141413 us; error 30 us; skew 500 ppm
I20260619 02:44:55.141839 13739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:55 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837095, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:55.144739 13739 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:44:55.146102 13739 webserver.cc:460] Webserver started at http://127.7.246.126:45453/ using document root <none> and password file <none>
I20260619 02:44:55.146781 13739 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:55.146874 13739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:55.147130 13739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:55.150059 13739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "b5a5393d927d4070988bf8b1e8d4a23b"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "a846788dcbcb57e3673d660176b7ad1f"
server_key_iv: "d96ae09305e3e83fd7f7593f5c13e33f"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.150753 13739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "b5a5393d927d4070988bf8b1e8d4a23b"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "a846788dcbcb57e3673d660176b7ad1f"
server_key_iv: "d96ae09305e3e83fd7f7593f5c13e33f"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.155982 13739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:44:55.159502 13754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:55.160873 13739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:55.161011 13739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "b5a5393d927d4070988bf8b1e8d4a23b"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "a846788dcbcb57e3673d660176b7ad1f"
server_key_iv: "d96ae09305e3e83fd7f7593f5c13e33f"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.161187 13739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:55.198276 13739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:55.203001 13739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:55.203238 13739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:55.221951 13739 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:38967
I20260619 02:44:55.221995 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:38967 every 8 connection(s)
I20260619 02:44:55.223335 13739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:55.227082 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:55.227437 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13739
I20260619 02:44:55.227671 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:55.228266 8153 external_mini_cluster.cc:1428] Setting key 826c52a7e1e17dc94d174c2b5c9d8735
I20260619 02:44:55.240803 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b: Bootstrap starting.
Jun 19 02:44:55 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:55.244235 13806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:55.245266 13806 log.cc:824] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:55.248513 13806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b: No bootstrap required, opened a new log
I20260619 02:44:55.251693 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:55.230172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.7.246.126:38967)
0619 02:44:55.230898 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:55.230909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:55.230946 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:44:55.232882 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:55.232904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:55.233309 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:55.233734 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:55.235364 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.236604 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.238448 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.239109 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.239238 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:55.243681 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:55.243688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:55.243702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:55.243736 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:55.247146 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:55.247805 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:55.247816 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:55.247826 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:55.248027 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:55.248444 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:55.248449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:55.248452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:55.249149 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:55.249449 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:55.249845 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 02:44:55.250278 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":336,"threads_started":1}
I20260619 02:44:55.257900 13806 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } }
I20260619 02:44:55.258301 13806 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:55.258381 13806 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a5393d927d4070988bf8b1e8d4a23b, State: Initialized, Role: FOLLOWER
I20260619 02:44:55.258961 13806 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } }
I20260619 02:44:55.259161 13806 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:55.259311 13806 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:55.259445 13806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:55.261938 13806 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } }
I20260619 02:44:55.262379 13806 leader_election.cc:302] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a5393d927d4070988bf8b1e8d4a23b; no voters:
I20260619 02:44:55.262902 13806 leader_election.cc:288] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:55.263062 13811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:55.263741 13811 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [term 1 LEADER]: Becoming Leader. State: Replica: b5a5393d927d4070988bf8b1e8d4a23b, State: Running, Role: LEADER
I20260619 02:44:55.264319 13811 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } }
I20260619 02:44:55.265130 13806 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:55.267127 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } } }
I20260619 02:44:55.267546 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:55.268028 13813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a5393d927d4070988bf8b1e8d4a23b. Latest consensus state: current_term: 1 leader_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a5393d927d4070988bf8b1e8d4a23b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38967 } } }
I20260619 02:44:55.268330 13813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:55.274202 13820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:55.278821 13820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:55.286871 13820 catalog_manager.cc:1347] Generated new cluster ID: 9dc902d08273418385266bf3610e0bc8
I20260619 02:44:55.286994 13820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:55.303879 13820 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:55.305311 13820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:55.311568 13820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a5393d927d4070988bf8b1e8d4a23b: Generated new TSK 0
I20260619 02:44:55.312377 13820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:44:55.369906 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:55.696110 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:55.696455 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:55.696640 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:55.704021 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:55.704180 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:55.704377 13832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:55.714228 13832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:55.715523 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:55.716841 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:55.729697 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:55.729748 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:55.729774 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:55.730540 13832 server_base.cc:1034] running on GCE node
I20260619 02:44:55.731065 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:55.731933 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:55.733139 13832 hybrid_clock.cc:648] HybridClock initialized: now 1781837095733077 us; error 55 us; skew 500 ppm
I20260619 02:44:55.733480 13832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:55 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837095, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:55.736486 13832 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:44:55.738027 13832 webserver.cc:460] Webserver started at http://127.7.246.65:42427/ using document root <none> and password file <none>
I20260619 02:44:55.738758 13832 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:55.738910 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:55.739226 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:55.742327 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "2784b53f44ff4f90b4802f02b3c17da5"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "18014721e90a2b453b547fd3dd494c57"
server_key_iv: "9adb6247b67a30178eb15973ab67b73b"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.743148 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "2784b53f44ff4f90b4802f02b3c17da5"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "18014721e90a2b453b547fd3dd494c57"
server_key_iv: "9adb6247b67a30178eb15973ab67b73b"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.748247 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:55.751977 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:55.753397 13832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:55.753631 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "2784b53f44ff4f90b4802f02b3c17da5"
format_stamp: "Formatted at 2026-06-19 02:44:55 on dist-test-slave-9tkz"
server_key: "18014721e90a2b453b547fd3dd494c57"
server_key_iv: "9adb6247b67a30178eb15973ab67b73b"
server_key_version: "encryptionkey@0"
I20260619 02:44:55.753844 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:55.773240 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:55.777441 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:55.777752 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:55.778750 13832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:55.780110 13832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:55.780216 13832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:55.780376 13832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:55.780452 13832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:55.815056 13832 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:35867
I20260619 02:44:55.815102 13959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:35867 every 8 connection(s)
I20260619 02:44:55.816287 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:55.821978 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13832
I20260619 02:44:55.822212 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:55.822853 8153 external_mini_cluster.cc:1428] Setting key 322b6d0bc320016f117e55f9f763667d
Jun 19 02:44:55 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837095, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:55.838523 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:55.818771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:49401 (local address 127.7.246.126:38967)
0619 02:44:55.819199 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:55.819203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:55.820180 (+ 977us) server_negotiation.cc:407] Connection header received
0619 02:44:55.822165 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:55.822175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:55.822239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:55.822370 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:55.823973 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.824997 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.827650 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.828141 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.828275 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:55.832670 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:55.832676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:55.832679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:55.832709 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:55.835242 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:55.836194 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:55.836199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:55.836201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:55.836311 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:55.836799 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:55.836802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:55.836804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:55.837033 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:55.837148 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:55.838128 (+ 980us) server_negotiation.cc:299] Negotiation successful
0619 02:44:55.838284 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
I20260619 02:44:55.840128 13963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:55.819328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:38967 (local address 127.7.246.65:49401)
0619 02:44:55.820007 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:55.820054 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:55.821845 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:55.822683 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:55.822691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:55.823266 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:55.823809 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:55.823826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.825196 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.825201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:55.827481 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:55.827499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:55.828395 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:55.828403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:55.828745 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:55.829534 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:55.829610 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:55.832420 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:55.835392 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:55.835403 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:55.835423 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:55.836065 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:55.836419 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:55.836426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:55.836432 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:55.836675 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:55.837173 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:55.837183 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:55.837692 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 02:44:55.838016 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 02:44:55.838637 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260619 02:44:55.841832 13960 heartbeater.cc:347] Connected to a master server at 127.7.246.126:38967
I20260619 02:44:55.842165 13960 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:55.842890 13960 heartbeater.cc:511] Master 127.7.246.126:38967 requested a full tablet report, sending...
I20260619 02:44:55.844939 13771 ts_manager.cc:194] Registered new tserver with Master: 2784b53f44ff4f90b4802f02b3c17da5 (127.7.246.65:35867)
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:44:55.846560 13771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:49401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:44:55.872310 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:56.185317 13966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:56.185642 13966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:56.185737 13966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:56.192647 13966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:56.192746 13966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:56.192857 13966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:44:56.202317 13966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:56.203508 13966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:56.204895 13966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:56.217633 13971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:56.217777 13974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:56.217777 13972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:56.218568 13966 server_base.cc:1034] running on GCE node
I20260619 02:44:56.219184 13966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:56.220201 13966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:56.221503 13966 hybrid_clock.cc:648] HybridClock initialized: now 1781837096221459 us; error 49 us; skew 500 ppm
I20260619 02:44:56.221944 13966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.225677 13966 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:44:56.230906 13966 webserver.cc:460] Webserver started at http://127.7.246.66:43101/ using document root <none> and password file <none>
I20260619 02:44:56.231599 13966 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:56.231745 13966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:56.232050 13966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:56.234920 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "46e60bd913de48d2a9b39ebeca85a767"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "7e63c3618859d3ea57cdfd7eb08956b7"
server_key_iv: "52712887231a73e7db1b5d0dcc4abdc0"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.235677 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "46e60bd913de48d2a9b39ebeca85a767"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "7e63c3618859d3ea57cdfd7eb08956b7"
server_key_iv: "52712887231a73e7db1b5d0dcc4abdc0"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.240659 13966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:44:56.244006 13981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.245231 13966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:44:56.245397 13966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "46e60bd913de48d2a9b39ebeca85a767"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "7e63c3618859d3ea57cdfd7eb08956b7"
server_key_iv: "52712887231a73e7db1b5d0dcc4abdc0"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.245637 13966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:56.273396 13966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:56.277529 13966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:56.277788 13966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:56.278678 13966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:56.279898 13966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:56.279968 13966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.280045 13966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:56.280086 13966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.312064 13966 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:34503
I20260619 02:44:56.312083 14093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:34503 every 8 connection(s)
I20260619 02:44:56.313328 13966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:44:56.321703 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 13966
I20260619 02:44:56.321905 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:44:56.322460 8153 external_mini_cluster.cc:1428] Setting key 5449e94ba273f9c07de7d7549aa37c9d
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:56.333597 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.315872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:60677 (local address 127.7.246.126:38967)
0619 02:44:56.316041 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.316045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.317142 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 02:44:56.318865 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.318869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.318931 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.319017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.320444 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.321135 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.323379 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.323863 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.324003 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.327753 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.327758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.327761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.327792 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.330357 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.331267 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.331271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.331273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.331402 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.331919 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.331926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.331934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.332186 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.332336 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.333004 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.333226 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:44:56.334720 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.316324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:38967 (local address 127.7.246.66:60677)
0619 02:44:56.316991 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:56.317030 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:56.318632 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:56.319211 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:56.319219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:56.319725 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:56.320306 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:56.320323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.321274 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.321281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:56.323253 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:56.323265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.324093 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.324101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:56.324406 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.325298 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:56.325329 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:56.327534 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:56.330502 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:56.330511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:56.330529 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:56.331131 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:56.331533 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:56.331538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:56.331542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:56.331778 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:56.332321 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:56.332328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:56.332661 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 02:44:56.332909 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 02:44:56.333425 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":266,"threads_started":1}
I20260619 02:44:56.336083 14094 heartbeater.cc:347] Connected to a master server at 127.7.246.126:38967
I20260619 02:44:56.336376 14094 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:56.337034 14094 heartbeater.cc:511] Master 127.7.246.126:38967 requested a full tablet report, sending...
I20260619 02:44:56.338585 13771 ts_manager.cc:194] Registered new tserver with Master: 46e60bd913de48d2a9b39ebeca85a767 (127.7.246.66:34503)
I20260619 02:44:56.339869 13771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:60677
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:44:56.368609 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:56.689417 14099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:56.689711 14099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:56.689880 14099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:56.697223 14099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:56.697333 14099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:56.697515 14099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:44:56.707458 14099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:56.708837 14099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:56.710145 14099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:56.723359 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:56.723367 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:56.723620 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:56.724072 14099 server_base.cc:1034] running on GCE node
I20260619 02:44:56.724757 14099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:56.725772 14099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:56.726997 14099 hybrid_clock.cc:648] HybridClock initialized: now 1781837096726953 us; error 59 us; skew 500 ppm
I20260619 02:44:56.727306 14099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.735982 14099 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:44:56.737380 14099 webserver.cc:460] Webserver started at http://127.7.246.67:42357/ using document root <none> and password file <none>
I20260619 02:44:56.738126 14099 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:56.738229 14099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:56.738466 14099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:56.741314 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "efac204beb3249f18ed1db858e943873"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "0d742c6edf038d57cc6390b6b3182a41"
server_key_iv: "dc45c19405bce815bc1481c0c801a31e"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.742108 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "efac204beb3249f18ed1db858e943873"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "0d742c6edf038d57cc6390b6b3182a41"
server_key_iv: "dc45c19405bce815bc1481c0c801a31e"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.747056 14099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:44:56.750753 14114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.752185 14099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:44:56.752358 14099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "efac204beb3249f18ed1db858e943873"
format_stamp: "Formatted at 2026-06-19 02:44:56 on dist-test-slave-9tkz"
server_key: "0d742c6edf038d57cc6390b6b3182a41"
server_key_iv: "dc45c19405bce815bc1481c0c801a31e"
server_key_version: "encryptionkey@0"
I20260619 02:44:56.752554 14099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:56.765780 14099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:56.770249 14099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:56.770565 14099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:56.771579 14099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:56.772874 14099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:56.772950 14099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.773082 14099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:56.773135 14099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:56.807396 14099 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:42961
I20260619 02:44:56.807454 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:42961 every 8 connection(s)
I20260619 02:44:56.808709 14099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:44:56.817610 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14099
I20260619 02:44:56.817821 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:44:56.818392 8153 external_mini_cluster.cc:1428] Setting key 275e0644f529a77de649ba9c9932006b
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:44:56.829676 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.811338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:43877 (local address 127.7.246.126:38967)
0619 02:44:56.811561 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.811566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.812956 (+ 1390us) server_negotiation.cc:407] Connection header received
0619 02:44:56.815001 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.815008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.815101 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.815233 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.816756 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.817860 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.819864 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.820286 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.820438 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.823721 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.823726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.823729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.823760 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.826688 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.827426 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.827433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.827437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.827557 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.827972 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.827977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.827981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.828242 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.828391 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.829079 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.829299 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 02:44:56.830616 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.811833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:38967 (local address 127.7.246.67:43877)
0619 02:44:56.812772 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:56.812818 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:56.814661 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:56.815483 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:56.815491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:56.816009 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:56.816572 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:56.816589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.818017 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.818025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:56.819712 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:56.819723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.820527 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.820535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:56.820758 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.821404 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:56.821426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:56.823519 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:56.826864 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:56.826873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:56.826891 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:56.827300 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:56.827668 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:56.827673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:56.827676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:56.827858 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:56.828380 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:56.828387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:56.828722 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 02:44:56.828988 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 02:44:56.829522 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":349,"threads_started":1}
I20260619 02:44:56.831974 14227 heartbeater.cc:347] Connected to a master server at 127.7.246.126:38967
I20260619 02:44:56.832336 14227 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:56.833249 14227 heartbeater.cc:511] Master 127.7.246.126:38967 requested a full tablet report, sending...
I20260619 02:44:56.834728 13771 ts_manager.cc:194] Registered new tserver with Master: efac204beb3249f18ed1db858e943873 (127.7.246.67:42961)
I20260619 02:44:56.835458 13771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:43877
I20260619 02:44:56.846057 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:44:56.849726 13960 heartbeater.cc:503] Master 127.7.246.126:38967 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/test-admin.keytab.
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.874977 8153 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 02:44:56.880417 14234 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:56.878104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.7.246.65:35867)
0619 02:44:56.878721 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.878729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.878754 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:44:56.878812 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.878818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.879021 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.879157 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.880011 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":333,"threads_started":1}
W20260619 02:44:56.880823 8153 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.882499 8153 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.883977 8153 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:56.898895 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.885709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.7.246.65:35867)
0619 02:44:56.885908 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.885914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.886076 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:44:56.886213 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.886217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.886274 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.886354 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.887333 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.888279 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.889432 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.890048 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.890144 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.893642 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.893653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.893668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.893700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.896284 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.896884 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.896890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.896893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.896999 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.897330 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.897333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.897335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.897733 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.897933 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.898340 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.898582 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 02:44:56.904722 14234 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:56.902842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.7.246.65:35867)
0619 02:44:56.902997 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.903001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.903448 (+ 447us) server_negotiation.cc:407] Connection header received
0619 02:44:56.903545 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.903550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.903614 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.903715 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.904504 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53352: BlockingRecv error: recv got EOF from 127.0.0.1:53352 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 02:44:56.904983 8153 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.906545 8153 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.907783 8153 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:56.921237 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.909110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.7.246.65:35867)
0619 02:44:56.909299 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.909305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.909482 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:44:56.909740 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.909744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.909797 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.909912 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.910803 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.911490 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.912676 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.913114 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.913245 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.916539 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.916544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.916547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.916588 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.919266 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.919812 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.919816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.919818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.919941 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.920245 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.920248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.920250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.920482 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.920679 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.920874 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.921025 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:44:56.925180 14234 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:44:56.923749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.7.246.65:35867)
0619 02:44:56.923950 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.923956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.924091 (+ 135us) server_negotiation.cc:407] Connection header received
0619 02:44:56.924189 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.924193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.924265 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.924351 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.924979 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53364: BlockingRecv error: recv got EOF from 127.0.0.1:53364 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 02:44:56.925406 8153 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837047186258-8153-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.926872 8153 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:56.928148 8153 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:44:56 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:44:56.941298 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.929293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.7.246.65:35867)
0619 02:44:56.929483 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.929488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.929718 (+ 230us) server_negotiation.cc:407] Connection header received
0619 02:44:56.929836 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.929839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.929889 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.929958 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.930860 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.931541 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.932654 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.933040 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.933139 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.936691 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.936695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.936698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.936727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.939123 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.939666 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.939670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.939673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.939830 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.940119 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.940122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.940124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.940374 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.940593 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.940777 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.940962 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:44:56.942795 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13832
I20260619 02:44:56.954403 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:56.943538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.7.246.65:35867)
0619 02:44:56.943706 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:56.943710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:56.943940 (+ 230us) server_negotiation.cc:407] Connection header received
0619 02:44:56.944058 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:56.944062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:56.944110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:56.944181 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:56.945037 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.945759 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.946851 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:56.947302 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:56.947393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:56.949442 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:56.949449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:56.949453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:56.949491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:56.952009 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.952604 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.952607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.952610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.952729 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:56.953012 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:56.953015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:56.953017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:56.953248 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:56.953422 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:56.953798 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 02:44:56.954012 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:44:56.955013 13955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:57.037012 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13832
I20260619 02:44:57.052812 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13966
Jun 19 02:44:57 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:44:57.068533 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:57.053755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.7.246.66:34503)
0619 02:44:57.054269 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:57.054276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:57.054300 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:44:57.054356 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:57.054362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:57.054569 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:57.054689 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:57.055531 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:57.056600 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:57.058026 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:57.058570 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:57.058667 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:57.062733 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:57.062745 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:57.062759 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:57.062793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:57.065932 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:57.066651 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:57.066659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:57.066661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:57.066787 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:57.067091 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:57.067094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:57.067096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:57.067455 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:57.067748 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:57.067978 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 02:44:57.068218 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":270,"threads_started":1}
I20260619 02:44:57.069867 14089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:57.151273 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13966
I20260619 02:44:57.167840 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14099
Jun 19 02:44:57 dist-test-slave-9tkz krb5kdc[13726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:44:57.183746 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:57.169030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.7.246.67:42961)
0619 02:44:57.169755 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:57.169761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:57.169790 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:44:57.169848 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:57.169854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:57.170085 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:57.170232 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:57.171270 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:57.172295 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:57.173500 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:57.174203 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:57.174305 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:57.178305 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:57.178317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:57.178333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:57.178371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:57.181226 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:57.182001 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:57.182008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:57.182010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:57.182127 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:57.182444 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:57.182447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:57.182448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:57.182777 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:57.183021 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:57.183219 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:44:57.183406 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":360,"threads_started":1}
I20260619 02:44:57.185034 14222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:57.267450 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14099
I20260619 02:44:57.283124 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 13739
I20260619 02:44:57.284332 13801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:44:57.384153 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 13739
2026-06-19T02:44:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3897 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:44:57 dist-test-slave-9tkz krb5kdc[14254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:44:57 dist-test-slave-9tkz krb5kdc[14254](info): set up 2 sockets
Jun 19 02:44:57 dist-test-slave-9tkz krb5kdc[14254](info): commencing operation
krb5kdc: starting...
W20260619 02:44:58.482404 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:44:58 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:44:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab.
Entry for principal oryx/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab.
I20260619 02:44:58.687494 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36239
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab
--principal=oryx/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:59.000972 14267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:59.001255 14267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:59.001410 14267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:59.008560 14267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:44:59.008661 14267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:59.008754 14267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:59.008827 14267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:44:59.008903 14267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:44:59.018669 14267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab
--principal=oryx/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:59.019949 14267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:59.021252 14267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:59.032325 14275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:59.032279 14272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:59.032356 14273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:59.033595 14267 server_base.cc:1034] running on GCE node
I20260619 02:44:59.034315 14267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:59.035681 14267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:59.036882 14267 hybrid_clock.cc:648] HybridClock initialized: now 1781837099036840 us; error 31 us; skew 500 ppm
I20260619 02:44:59.037209 14267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:59 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837099, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:59.040189 14267 init.cc:377] Logged in from keytab as oryx/127.7.246.126@KRBTEST.COM (short username oryx)
I20260619 02:44:59.041667 14267 webserver.cc:460] Webserver started at http://127.7.246.126:38521/ using document root <none> and password file <none>
I20260619 02:44:59.042286 14267 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:59.042389 14267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:59.042625 14267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:59.045542 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "246a54d557974c13a38a4faee724048b"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "cde26c4d8337d6569138de5541c8e085"
server_key_iv: "02446a48600d3356fab3cbaa89b843ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.046274 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "246a54d557974c13a38a4faee724048b"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "cde26c4d8337d6569138de5541c8e085"
server_key_iv: "02446a48600d3356fab3cbaa89b843ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.051254 14267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 02:44:59.054507 14282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:59.055944 14267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:44:59.056075 14267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "246a54d557974c13a38a4faee724048b"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "cde26c4d8337d6569138de5541c8e085"
server_key_iv: "02446a48600d3356fab3cbaa89b843ba"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.056231 14267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:59.073437 14267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:59.077483 14267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:59.077780 14267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:59.095697 14267 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:36239
I20260619 02:44:59.095726 14333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:36239 every 8 connection(s)
I20260619 02:44:59.097587 14267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:44:59.101275 14334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:44:59.106597 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14267
I20260619 02:44:59.106851 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:44:59.107463 8153 external_mini_cluster.cc:1428] Setting key e7c84667a91dfc7cbb12f47f6be2caaf
I20260619 02:44:59.112841 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Bootstrap starting.
I20260619 02:44:59.116480 14334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Neither blocks nor log segments found. Creating new log.
I20260619 02:44:59.117705 14334 log.cc:824] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:44:59.120981 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: No bootstrap required, opened a new log
Jun 19 02:44:59 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:44:59.130530 14334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } }
I20260619 02:44:59.131024 14334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:44:59.131129 14334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246a54d557974c13a38a4faee724048b, State: Initialized, Role: FOLLOWER
I20260619 02:44:59.131795 14334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } }
I20260619 02:44:59.132084 14334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:44:59.132222 14334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:44:59.132395 14334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:44:59.132794 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:59.109613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.7.246.126:36239)
0619 02:44:59.110462 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:59.110478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:59.110540 (+ 62us) server_negotiation.cc:407] Connection header received
0619 02:44:59.112780 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:59.112808 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:59.113291 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:59.113774 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:59.115271 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.117241 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.119006 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.119985 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.120155 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:59.124302 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:59.124312 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:59.124332 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:59.124384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:59.128368 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:59.129066 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:59.129075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:59.129084 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:59.129281 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:59.129665 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:59.129672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:59.129690 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:59.130357 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:59.130675 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:59.130950 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:44:59.131382 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":353,"threads_started":1}
I20260619 02:44:59.135545 14334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } }
I20260619 02:44:59.136111 14334 leader_election.cc:302] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 246a54d557974c13a38a4faee724048b; no voters:
I20260619 02:44:59.136898 14334 leader_election.cc:288] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:44:59.137012 14339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:44:59.139700 14334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:44:59.144767 14339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 LEADER]: Becoming Leader. State: Replica: 246a54d557974c13a38a4faee724048b, State: Running, Role: LEADER
I20260619 02:44:59.146184 14339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } }
I20260619 02:44:59.150046 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } }
I20260619 02:44:59.150198 14341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246a54d557974c13a38a4faee724048b. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } }
I20260619 02:44:59.150566 14341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:59.150569 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:44:59.151558 14348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:44:59.158085 14348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:44:59.171096 14348 catalog_manager.cc:1347] Generated new cluster ID: 61838aebcd5246fe86c6a4b05f5c3e0e
I20260619 02:44:59.171262 14348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:44:59.194468 14348 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:44:59.196287 14348 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:44:59.202385 14348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Generated new TSK 0
I20260619 02:44:59.203174 14348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
I20260619 02:44:59.253890 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:44:59.567139 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:44:59.567431 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:44:59.567590 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:44:59.574551 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:44:59.574704 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:44:59.574869 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:44:59.584340 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:44:59.585582 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:44:59.586856 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:44:59.599159 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:59.599232 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:44:59.599253 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:44:59.600127 14360 server_base.cc:1034] running on GCE node
I20260619 02:44:59.600696 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:44:59.601593 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:44:59.602841 14360 hybrid_clock.cc:648] HybridClock initialized: now 1781837099602781 us; error 46 us; skew 500 ppm
I20260619 02:44:59.603175 14360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:44:59 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837099, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:44:59.606324 14360 init.cc:377] Logged in from keytab as oryx/127.7.246.65@KRBTEST.COM (short username oryx)
I20260619 02:44:59.607630 14360 webserver.cc:460] Webserver started at http://127.7.246.65:41747/ using document root <none> and password file <none>
I20260619 02:44:59.608294 14360 fs_manager.cc:359] Metadata directory not provided
I20260619 02:44:59.608443 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:44:59.608753 14360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:44:59.611680 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "e7478f303cc94ce6af89c965b493a3a0"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "4a5101e6a94efd5232ad9274c523a6d4"
server_key_iv: "e9c0e5cb33e150e6e4ad8502adc4de03"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.612421 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "e7478f303cc94ce6af89c965b493a3a0"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "4a5101e6a94efd5232ad9274c523a6d4"
server_key_iv: "e9c0e5cb33e150e6e4ad8502adc4de03"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.617188 14360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260619 02:44:59.620576 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:59.621837 14360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 02:44:59.622000 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "e7478f303cc94ce6af89c965b493a3a0"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "4a5101e6a94efd5232ad9274c523a6d4"
server_key_iv: "e9c0e5cb33e150e6e4ad8502adc4de03"
server_key_version: "encryptionkey@0"
I20260619 02:44:59.622208 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:44:59.639467 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:44:59.643617 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:44:59.643890 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:44:59.644796 14360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:44:59.646164 14360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:44:59.646236 14360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:59.646346 14360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:44:59.646409 14360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:44:59.676833 14360 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:39293
I20260619 02:44:59.676894 14487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:39293 every 8 connection(s)
I20260619 02:44:59.678176 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:44:59.684453 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14360
I20260619 02:44:59.684705 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:44:59.685295 8153 external_mini_cluster.cc:1428] Setting key 607b2bcc8364d7781887b85eef098cfe
Jun 19 02:44:59 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837099, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:44:59.698203 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:59.680679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:42241 (local address 127.7.246.126:36239)
0619 02:44:59.681099 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 02:44:59.681103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:44:59.681918 (+ 815us) server_negotiation.cc:407] Connection header received
0619 02:44:59.683721 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:44:59.683726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:44:59.683810 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:44:59.683925 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:44:59.685585 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.686385 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.688032 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.688797 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.688957 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:59.692223 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:44:59.692232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:44:59.692240 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:44:59.692284 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:44:59.695043 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:59.695950 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:59.695954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:59.695956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:59.696067 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:44:59.696523 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:44:59.696526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:44:59.696528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:44:59.696773 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:44:59.696886 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:44:59.697740 (+ 854us) server_negotiation.cc:299] Negotiation successful
0619 02:44:59.697937 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":240,"threads_started":1}
I20260619 02:44:59.699846 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:44:59.681052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:42241)
0619 02:44:59.681753 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 02:44:59.681796 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:44:59.683448 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:44:59.684167 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:44:59.684174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:44:59.684802 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:44:59.685358 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:59.685413 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.686537 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.686541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:59.687887 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:44:59.687898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:44:59.689002 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:44:59.689006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:44:59.689193 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:44:59.689914 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:44:59.689940 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:44:59.691984 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:44:59.695176 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:59.695188 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:59.695207 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:59.695826 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:59.696177 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:44:59.696184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:44:59.696190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:44:59.696421 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:44:59.696890 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:44:59.696900 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:44:59.697347 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 02:44:59.697672 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 02:44:59.698252 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
I20260619 02:44:59.701382 14488 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:44:59.701742 14488 heartbeater.cc:464] Registering TS with master...
I20260619 02:44:59.702548 14488 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:44:59.704598 14299 ts_manager.cc:194] Registered new tserver with Master: e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:44:59.706046 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:42241
WARNING: no policy specified for oryx/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
I20260619 02:44:59.732762 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:00.044653 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:00.044924 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:00.045096 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:00.056046 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:00.056173 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:00.056380 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:00.066298 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:00.067541 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:00.068822 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:00.081511 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:00.081625 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:00.081579 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:00.082741 14494 server_base.cc:1034] running on GCE node
I20260619 02:45:00.083283 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:00.084178 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:00.085392 14494 hybrid_clock.cc:648] HybridClock initialized: now 1781837100085351 us; error 25 us; skew 500 ppm
I20260619 02:45:00.085731 14494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:00 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837100, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:00.088747 14494 init.cc:377] Logged in from keytab as oryx/127.7.246.66@KRBTEST.COM (short username oryx)
I20260619 02:45:00.090145 14494 webserver.cc:460] Webserver started at http://127.7.246.66:40357/ using document root <none> and password file <none>
I20260619 02:45:00.090809 14494 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:00.090947 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:00.091245 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:00.094591 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "ba3f228c8d1c425184bc717e23396549"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "f37232643c8abe93e1ee6b0f85ec7aec"
server_key_iv: "7fe0d37e5797e06ea062b5560a067b69"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.095408 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "ba3f228c8d1c425184bc717e23396549"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "f37232643c8abe93e1ee6b0f85ec7aec"
server_key_iv: "7fe0d37e5797e06ea062b5560a067b69"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.101009 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:45:00.104461 14509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:00.105782 14494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:45:00.105940 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "ba3f228c8d1c425184bc717e23396549"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "f37232643c8abe93e1ee6b0f85ec7aec"
server_key_iv: "7fe0d37e5797e06ea062b5560a067b69"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.106127 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:00.121167 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:00.125052 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:00.125382 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:00.126430 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:00.127745 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:00.127847 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:00.127974 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:00.128044 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:00.161034 14494 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:39321
I20260619 02:45:00.161085 14621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:39321 every 8 connection(s)
I20260619 02:45:00.162367 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:00.171095 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14494
I20260619 02:45:00.171283 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:00.171705 8153 external_mini_cluster.cc:1428] Setting key d958184e16a094b9cbc44125afc650c6
Jun 19 02:45:00 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837100, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:00.183051 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:00.164785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:43269 (local address 127.7.246.126:36239)
0619 02:45:00.164963 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:00.164967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:00.166123 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 02:45:00.167863 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:00.167869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:00.167962 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:00.168073 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:00.169497 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.170265 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.172080 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.172518 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.172664 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:00.176491 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:00.176500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:00.176507 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:00.176552 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:00.180046 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:00.180762 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:00.180770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:00.180773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:00.180905 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:00.181328 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:00.181333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:00.181336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:00.181684 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:00.181831 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:00.182501 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 02:45:00.182719 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:45:00.184003 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:00.165196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:43269)
0619 02:45:00.165950 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:00.165991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:00.167603 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:00.168299 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:00.168307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:00.168783 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:00.169335 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:00.169352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.170493 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.170500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:00.171930 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:00.171942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.172753 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.172759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:00.172983 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:00.173695 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:00.173723 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:00.176283 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:00.180176 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:00.180183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:00.180196 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:00.180613 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:00.181025 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:00.181036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:00.181039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:00.181208 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:00.181803 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:00.181811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:00.182161 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 02:45:00.182421 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 02:45:00.182935 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":303,"threads_started":1}
I20260619 02:45:00.185614 14622 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:45:00.185943 14622 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:00.186671 14622 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:45:00.188191 14299 ts_manager.cc:194] Registered new tserver with Master: ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321)
I20260619 02:45:00.189523 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:43269
WARNING: no policy specified for oryx/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
I20260619 02:45:00.220677 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:00.544567 14627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:00.544870 14627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:00.544992 14627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:00.552204 14627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:00.552312 14627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:00.552423 14627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:00.562280 14627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:00.563519 14627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:00.564798 14627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:00.578222 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:00.578305 14635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:00.578496 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:00.579051 14627 server_base.cc:1034] running on GCE node
I20260619 02:45:00.579677 14627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:00.580636 14627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:00.581831 14627 hybrid_clock.cc:648] HybridClock initialized: now 1781837100581791 us; error 37 us; skew 500 ppm
I20260619 02:45:00.582304 14627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:00 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837100, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:00.586692 14627 init.cc:377] Logged in from keytab as oryx/127.7.246.67@KRBTEST.COM (short username oryx)
I20260619 02:45:00.588927 14627 webserver.cc:460] Webserver started at http://127.7.246.67:33037/ using document root <none> and password file <none>
I20260619 02:45:00.589701 14627 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:00.589824 14627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:00.590123 14627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:00.592995 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "ad410b9ad6704ad095daa5fbbcf3088c"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "351f46bb1a26c5565bad7c8b5499a5af"
server_key_iv: "e5fa4ea8bd88c193da4070ebd671c80d"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.593811 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "ad410b9ad6704ad095daa5fbbcf3088c"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "351f46bb1a26c5565bad7c8b5499a5af"
server_key_iv: "e5fa4ea8bd88c193da4070ebd671c80d"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.598878 14627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 02:45:00.602262 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:00.603652 14627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:00.603819 14627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "ad410b9ad6704ad095daa5fbbcf3088c"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "351f46bb1a26c5565bad7c8b5499a5af"
server_key_iv: "e5fa4ea8bd88c193da4070ebd671c80d"
server_key_version: "encryptionkey@0"
I20260619 02:45:00.604000 14627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:00.622406 14627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:00.626796 14627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:00.627084 14627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:00.627985 14627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:00.629330 14627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:00.629405 14627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:00.629619 14627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:00.629680 14627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 02:45:00.660717 14627 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:46829
I20260619 02:45:00.660748 14754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:46829 every 8 connection(s)
I20260619 02:45:00.662125 14627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:00.670944 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14627
I20260619 02:45:00.671170 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:00.671581 8153 external_mini_cluster.cc:1428] Setting key 1f356c91300cef7c718756a17eb38f85
Jun 19 02:45:00 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837100, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:00.682770 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:00.664819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:56867 (local address 127.7.246.126:36239)
0619 02:45:00.664983 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:00.664987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:00.666238 (+ 1251us) server_negotiation.cc:407] Connection header received
0619 02:45:00.667996 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:00.668001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:00.668099 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:00.668203 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:00.669790 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.670535 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.672612 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.673274 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.673390 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:00.677249 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:00.677254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:00.677257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:00.677293 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:00.679835 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:00.680592 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:00.680599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:00.680607 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:00.680712 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:00.681124 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:00.681128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:00.681132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:00.681397 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:00.681582 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:00.682213 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 02:45:00.682421 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:45:00.683676 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:00.665317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:56867)
0619 02:45:00.666082 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:00.666125 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:00.667738 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:00.668453 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:00.668462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:00.669007 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:00.669634 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:00.669653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.670769 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.670774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:00.672455 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:00.672493 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:00.673609 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:00.673622 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:00.673945 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:00.674688 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:00.674724 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:00.677025 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:00.679973 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:00.679982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:00.679997 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:00.680467 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:00.680818 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:00.680824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:00.680828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:00.681002 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:00.681521 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:00.681529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:00.681873 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 02:45:00.682137 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 02:45:00.682641 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":323,"threads_started":1}
I20260619 02:45:00.684932 14755 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:45:00.685238 14755 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:00.686074 14755 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:45:00.687522 14299 ts_manager.cc:194] Registered new tserver with Master: ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:00.688324 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:56867
I20260619 02:45:00.699774 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:45:00.708763 14488 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.7.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.125.keytab.
Entry for principal oryx/127.7.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.125.keytab.
I20260619 02:45:00.748368 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:45621
--webserver_interface=127.7.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.125.keytab
--principal=oryx/127.7.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36239,127.7.246.125:45621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:01.080452 14760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:01.080760 14760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:01.080888 14760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:01.088500 14760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:01.088618 14760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:01.088742 14760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:01.088856 14760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:01.088943 14760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:01.099028 14760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36239,127.7.246.125:45621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.125.keytab
--principal=oryx/127.7.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:45621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
W20260619 02:45:01.099422 14760 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.7.246.126:36239,127.7.246.125:45621'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:45:01.100490 14760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:01.101938 14760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:01.113961 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.114029 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.114442 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:01.115242 14760 server_base.cc:1034] running on GCE node
I20260619 02:45:01.116052 14760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:01.117533 14760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:01.118820 14760 hybrid_clock.cc:648] HybridClock initialized: now 1781837101118759 us; error 42 us; skew 500 ppm
I20260619 02:45:01.119170 14760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:01.122316 14760 init.cc:377] Logged in from keytab as oryx/127.7.246.125@KRBTEST.COM (short username oryx)
I20260619 02:45:01.123843 14760 webserver.cc:460] Webserver started at http://127.7.246.125:43817/ using document root <none> and password file <none>
I20260619 02:45:01.124614 14760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:01.124744 14760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:01.125061 14760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:01.128229 14760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/instance:
uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "dfc7940ac5d88439f31431b6528d7c9b"
server_key_iv: "c849e19cdf107d7e0528fcb922a17a4b"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.129019 14760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal/instance:
uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "dfc7940ac5d88439f31431b6528d7c9b"
server_key_iv: "c849e19cdf107d7e0528fcb922a17a4b"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.134097 14760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:45:01.137521 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:01.138907 14760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:01.139087 14760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "dfc7940ac5d88439f31431b6528d7c9b"
server_key_iv: "c849e19cdf107d7e0528fcb922a17a4b"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.139308 14760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:01.152216 14760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:01.156157 14760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:01.156448 14760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:01.175330 14760 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.125:45621
I20260619 02:45:01.175350 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.125:45621 every 8 connection(s)
I20260619 02:45:01.176718 14760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
I20260619 02:45:01.179612 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14760
I20260619 02:45:01.180840 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:01.188881 14827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:45:01.192795 14622 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.125@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:01.202833 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.181774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.7.246.125:45621)
0619 02:45:01.182509 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.182522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.182561 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:45:01.184164 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.184185 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.184586 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.185023 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.186566 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.187854 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.189077 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.189921 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.190153 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.195183 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.195190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.195200 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.195237 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.198742 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.199478 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.199487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.199502 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.199700 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.200053 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.200059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.200062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.200688 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.200902 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.201376 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.201858 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":313,"threads_started":1}
I20260619 02:45:01.207630 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.191438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43954)
0619 02:45:01.192193 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.192241 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.192423 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.193347 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.193384 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.194228 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.194578 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.194588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.195726 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.195730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.197590 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.197606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.198299 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.198306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.198506 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.199146 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.199167 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.201873 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.204859 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.204872 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.204878 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.205333 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.205804 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.205809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.205813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.205942 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.206496 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.206503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.206719 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.206897 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.207229 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":302,"threads_started":1}
I20260619 02:45:01.207613 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.191572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.7.246.126:36239)
0619 02:45:01.192951 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.192957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.192975 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:45:01.193042 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.193046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.193128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.193240 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.194727 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.195590 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.197726 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.198113 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.198206 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.202055 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.202060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.202063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.202093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.204722 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.205461 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.205471 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.205473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.205692 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.206072 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.206077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.206080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.206373 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.206533 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.206992 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.207218 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1140,"threads_started":1}
I20260619 02:45:01.209337 14827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.125@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
I20260619 02:45:01.220775 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.209958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45000)
0619 02:45:01.210176 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.210194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.210285 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.210627 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.210630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.210905 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.211057 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.211065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.211965 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.211968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.212808 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.212817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.213427 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.213431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.213616 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.214315 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.214332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.216245 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.218855 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.218859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.218862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.219221 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.219571 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.219574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.219577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.219688 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.220112 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.220115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.220236 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.220338 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.220541 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:45:01.221076 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.210230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.7.246.125:45621)
0619 02:45:01.210370 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.210374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.210388 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:45:01.210436 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.210439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.210487 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.210576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.211169 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.211843 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.212919 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.213310 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.213400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.216400 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.216405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.216408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.216435 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.218736 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.219323 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.219327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.219329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.219470 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.219773 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.219776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.219778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.220022 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.220124 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.220425 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.220660 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":46}
I20260619 02:45:01.225600 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Bootstrap starting.
I20260619 02:45:01.228804 14827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:01.229894 14827 log.cc:824] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:01.232532 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: No bootstrap required, opened a new log
I20260619 02:45:01.239354 14827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } }
I20260619 02:45:01.239684 14827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:01.239781 14827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550ed9b2244d4d0ebcc10a7c63e7df82, State: Initialized, Role: FOLLOWER
I20260619 02:45:01.240453 14827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } }
I20260619 02:45:01.241268 14834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } } }
I20260619 02:45:01.241670 14834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:01.242373 14827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:45:01.251602 14845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:45:01.251811 14845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:45:01.253335 14760 master_runner.cc:428] Detected that this master 550ed9b2244d4d0ebcc10a7c63e7df82 is joining an existing cluster
I20260619 02:45:01.253433 14760 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.246.125:45621
I20260619 02:45:01.273910 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.257625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.7.246.126:36239)
0619 02:45:01.257826 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.257832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.258407 (+ 575us) server_negotiation.cc:407] Connection header received
0619 02:45:01.258586 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.258592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.258685 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.258795 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.260216 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.261338 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.263482 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.264142 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.264289 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.266958 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.266967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.266973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.267017 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.270608 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.271548 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.271557 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.271562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.271727 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.272211 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.272217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.272220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.272568 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.272720 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.273284 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.273569 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:45:01.274026 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.257640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43958)
0619 02:45:01.258296 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.258316 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.258462 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.258841 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.258848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.259582 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.260069 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.260086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.261488 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.261496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.263321 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.263338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.264351 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.264359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.264561 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.265278 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.265308 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.266669 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.270777 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.270782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.270786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.271371 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.271850 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.271869 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.271874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.272080 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.272703 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.272709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.272958 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.273156 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.273539 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":28,"thread_start_us":3928,"threads_started":1}
I20260619 02:45:01.284550 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.262320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.7.246.125:45621)
0619 02:45:01.262540 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.262547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.270260 (+ 7713us) server_negotiation.cc:407] Connection header received
0619 02:45:01.270435 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.270441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.270497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.270602 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.271598 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.272575 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.274513 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.274990 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.275090 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.277390 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.277395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.277398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.277440 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.281997 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.282629 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.282636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.282639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.282755 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.283109 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.283117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.283120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.283400 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.283573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.283875 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.284219 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":63}
I20260619 02:45:01.284584 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.262076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45010)
0619 02:45:01.270155 (+ 8079us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.270171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.270313 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.270614 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.270619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.271158 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.271436 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.271444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.272775 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.272779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.274374 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.274387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.275349 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.275355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.275509 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.276178 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.276201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.277210 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.282148 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.282152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.282156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.282515 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.282859 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.282865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.282868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.283013 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.283521 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.283532 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.283666 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.283799 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.284097 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7956,"mutex_wait_us":4,"thread_start_us":7743,"threads_started":1}
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837099, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.126@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.7.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.124@KRBTEST.COM" created.
I20260619 02:45:01.300133 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.281529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.7.246.125:45621)
0619 02:45:01.284933 (+ 3404us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.284938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.284954 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:01.285058 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.285063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.285185 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.285284 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.286983 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.287689 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.288986 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.289496 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.289686 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.293539 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.293589 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.293594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.293635 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.296846 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.297750 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.297761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.297771 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.297902 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.298394 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.298398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.298401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.298888 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.299047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.299494 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.299726 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3286,"thread_start_us":8829,"threads_started":1}
I20260619 02:45:01.300171 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.281017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45022)
0619 02:45:01.281709 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.281750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.281943 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.285571 (+ 3628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.285582 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.286530 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.286819 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.286830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.287826 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.287831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.288840 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.288852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.289715 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.289721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.289930 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.290728 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.290763 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.293332 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.297038 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.297054 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.297064 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.297613 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.298099 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.298106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.298110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.298256 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.299022 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.299031 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.299215 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.299392 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.299744 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":813,"threads_started":1}
I20260619 02:45:01.303431 14299 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.246.125:45621: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
member_type: NON_VOTER
last_known_addr {
host: "127.7.246.125"
port: 45621
}
attrs {
promote: true
}
}
dest_uuid: "246a54d557974c13a38a4faee724048b"
cas_config_opid_index: -1
I20260619 02:45:01.304437 14299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } }
I20260619 02:45:01.322899 14858 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } }
I20260619 02:45:01.324301 14802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:01.327584 14858 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.246.125:45621
I20260619 02:45:01.328423 14859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } }
I20260619 02:45:01.328788 14760 master.cc:537] Master@127.7.246.125:45621 shutting down...
I20260619 02:45:01.328857 14859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:01.330390 14802 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Refusing update from remote peer 246a54d557974c13a38a4faee724048b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:45:01.331455 14859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Connected to new peer: Peer: permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.124.keytab.
Entry for principal oryx/127.7.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.124.keytab.
I20260619 02:45:01.335207 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:44117
--webserver_interface=127.7.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.124.keytab
--principal=oryx/127.7.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:45:01.338625 14834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:01.338920 14834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } }
I20260619 02:45:01.340189 14834 leader_election.cc:288] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 246a54d557974c13a38a4faee724048b (127.7.246.126:36239)
I20260619 02:45:01.340874 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "246a54d557974c13a38a4faee724048b" is_pre_election: true
I20260619 02:45:01.342082 14776 leader_election.cc:302] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 550ed9b2244d4d0ebcc10a7c63e7df82; no voters: 246a54d557974c13a38a4faee724048b
W20260619 02:45:01.342559 14283 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 02:45:01.342733 14834 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260619 02:45:01.345474 14283 consensus_peers.cc:487] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b -> Peer 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125:45621): Couldn't send request to peer 550ed9b2244d4d0ebcc10a7c63e7df82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:45:01.349963 14760 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:01.350381 14760 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:01.350577 14760 tablet_replica.cc:330] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: stopping tablet replica
I20260619 02:45:01.357956 14760 master.cc:552] Master@127.7.246.125:45621 shutdown complete.
I20260619 02:45:01.358075 14760 master_runner.cc:305] Clearing existing system tablet
I20260619 02:45:01.358615 14760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:01.358724 14760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:01.362154 14760 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:45:01.363919 14864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:01.364324 14760 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:01.364627 14760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 02:45:01.364723 14760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "dfc7940ac5d88439f31431b6528d7c9b"
server_key_iv: "c849e19cdf107d7e0528fcb922a17a4b"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.370272 14760 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:01.377513 14760 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:45:01.377748 14760 log.cc:1196] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:45:01.378265 14760 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Deleting consensus metadata
I20260619 02:45:01.378587 14760 master_runner.cc:315] Copying system tablet from 127.7.246.126:36239
I20260619 02:45:01.381222 14760 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: tablet copy: Beginning tablet copy session from remote peer at address 127.7.246.126:36239
I20260619 02:45:01.402194 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.382193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.7.246.126:36239)
0619 02:45:01.382388 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.382395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.386236 (+ 3841us) server_negotiation.cc:407] Connection header received
0619 02:45:01.386483 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.386490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.386587 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.386695 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.387927 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.389121 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.391164 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.391836 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.391974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.394586 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.394593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.394598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.394636 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.398552 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.399615 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.399621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.399625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.399774 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.400323 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.400329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.400332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.400685 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.400833 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.401513 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.401798 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:45:01.402516 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.382097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43968)
0619 02:45:01.386047 (+ 3950us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.386071 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.386295 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.386777 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.386783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.387387 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.387735 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.387749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.389306 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.389312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.390998 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.391014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.392022 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.392029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.392243 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.392996 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.393027 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.394360 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.398723 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.398731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.398738 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.399461 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.399917 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.399923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.399929 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.400173 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.400938 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.400945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.401168 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.401388 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.401868 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3799,"thread_start_us":3638,"threads_started":1}
I20260619 02:45:01.403548 14319 tablet_copy_service.cc:140] P 246a54d557974c13a38a4faee724048b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 550ed9b2244d4d0ebcc10a7c63e7df82 ({username='oryx', principal='oryx/127.7.246.125@KRBTEST.COM'} at 127.0.0.1:43968)
I20260619 02:45:01.403813 14319 tablet_copy_service.cc:161] P 246a54d557974c13a38a4faee724048b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 550ed9b2244d4d0ebcc10a7c63e7df82 at {username='oryx', principal='oryx/127.7.246.125@KRBTEST.COM'} at 127.0.0.1:43968: session id = 550ed9b2244d4d0ebcc10a7c63e7df82-00000000000000000000000000000000
I20260619 02:45:01.408636 14319 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:45:01.411012 14760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:45:01.418568 14760 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: tablet copy: Starting download of 0 data blocks...
I20260619 02:45:01.418932 14760 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: tablet copy: Starting download of 1 WAL segments...
I20260619 02:45:01.421993 14760 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:45:01.427052 14319 tablet_copy_service.cc:342] P 246a54d557974c13a38a4faee724048b: Request end of tablet copy session 550ed9b2244d4d0ebcc10a7c63e7df82-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.246.125@KRBTEST.COM'} at 127.0.0.1:43968
I20260619 02:45:01.427265 14319 tablet_copy_service.cc:433] P 246a54d557974c13a38a4faee724048b: ending tablet copy session 550ed9b2244d4d0ebcc10a7c63e7df82-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 550ed9b2244d4d0ebcc10a7c63e7df82
I20260619 02:45:01.439576 14760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:01.442525 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.443127 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.453665 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:01.453943 14760 server_base.cc:1034] running on GCE node
I20260619 02:45:01.454514 14760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:01.455293 14760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:01.456593 14760 hybrid_clock.cc:648] HybridClock initialized: now 1781837101456541 us; error 44 us; skew 500 ppm
I20260619 02:45:01.456836 14760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:01.459265 14760 init.cc:377] Logged in from keytab as oryx/127.7.246.125@KRBTEST.COM (short username oryx)
I20260619 02:45:01.460736 14760 webserver.cc:460] Webserver started at http://127.7.246.125:44097/ using document root <none> and password file <none>
I20260619 02:45:01.461153 14760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:01.461269 14760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:01.464892 14760 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:45:01.467084 14881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:01.467590 14760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:45:01.467767 14760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "dfc7940ac5d88439f31431b6528d7c9b"
server_key_iv: "c849e19cdf107d7e0528fcb922a17a4b"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.468008 14760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:01.478394 14760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:01.483119 14760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:01.511057 14760 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.125:45621
I20260619 02:45:01.511087 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.125:45621 every 8 connection(s)
I20260619 02:45:01.512315 14760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
I20260619 02:45:01.518142 14933 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:45:01.522377 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Bootstrap starting.
I20260619 02:45:01.539665 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:45:01.540261 14933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: Bootstrap complete.
I20260619 02:45:01.541385 14933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } }
I20260619 02:45:01.541688 14933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 550ed9b2244d4d0ebcc10a7c63e7df82, State: Initialized, Role: LEARNER
I20260619 02:45:01.541965 14933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } }
I20260619 02:45:01.543058 14933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:01.542892 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } }
I20260619 02:45:01.543349 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:45:01.553675 14947 catalog_manager.cc:1259] Loaded cluster ID: 61838aebcd5246fe86c6a4b05f5c3e0e
I20260619 02:45:01.553805 14947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: loading cluster ID for follower catalog manager: success
I20260619 02:45:01.558157 14947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: acquiring CA information for follower catalog manager: success
I20260619 02:45:01.561298 14947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:45:01.690773 14755 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
I20260619 02:45:01.726560 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.718516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45036)
0619 02:45:01.718751 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.718771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.718894 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.719809 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.719814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.719845 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:01.720058 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.720072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.722022 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.722028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.724378 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.724395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.725709 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.725715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.725946 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.726016 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.726119 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.726285 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 02:45:01.726665 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.718748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.7.246.125:45621)
0619 02:45:01.719426 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.719431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.719454 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:01.719543 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.719548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.719624 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.719744 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:01.720264 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.721858 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.724534 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.725507 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.725665 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.725848 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.726210 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.726381 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":373,"threads_started":1}
I20260619 02:45:01.727675 14908 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 02:45:01.728319 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246a54d557974c13a38a4faee724048b. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } }
I20260619 02:45:01.728575 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: LEARNER
W20260619 02:45:01.744165 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:01.744536 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:01.744644 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:01.752416 14861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:01.752532 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:01.752583 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:01.752638 14861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:01.752684 14861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:01.762864 14861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.124.keytab
--principal=oryx/127.7.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:01.764091 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:01.765292 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:01.777319 14954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.777314 14953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:01.777575 14956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:01.778223 14861 server_base.cc:1034] running on GCE node
I20260619 02:45:01.779034 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:01.780403 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:01.781647 14861 hybrid_clock.cc:648] HybridClock initialized: now 1781837101781599 us; error 75 us; skew 500 ppm
I20260619 02:45:01.782003 14861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:01.790270 14861 init.cc:377] Logged in from keytab as oryx/127.7.246.124@KRBTEST.COM (short username oryx)
I20260619 02:45:01.791826 14861 webserver.cc:460] Webserver started at http://127.7.246.124:42223/ using document root <none> and password file <none>
I20260619 02:45:01.792522 14861 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:01.792668 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:01.792974 14861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:01.795897 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/instance:
uuid: "22aeb734eae949f1aaee5d2ae407820a"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "d33cb640799cc2d3e1f5d1bc7a93406c"
server_key_iv: "86dd8c3b950412db09de631879413992"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.796665 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal/instance:
uuid: "22aeb734eae949f1aaee5d2ae407820a"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "d33cb640799cc2d3e1f5d1bc7a93406c"
server_key_iv: "86dd8c3b950412db09de631879413992"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.802295 14861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:45:01.805518 14963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:01.806932 14861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:45:01.807125 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "22aeb734eae949f1aaee5d2ae407820a"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "d33cb640799cc2d3e1f5d1bc7a93406c"
server_key_iv: "86dd8c3b950412db09de631879413992"
server_key_version: "encryptionkey@0"
I20260619 02:45:01.807394 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:01.823467 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:01.828086 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:01.828315 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:01.847877 14861 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.124:44117
I20260619 02:45:01.847895 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.124:44117 every 8 connection(s)
I20260619 02:45:01.849287 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
I20260619 02:45:01.852126 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 14861
I20260619 02:45:01.853152 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:01.862936 15015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.124@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:01.875380 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.854288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.7.246.124:44117)
0619 02:45:01.855005 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.855019 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.855064 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:45:01.856886 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.856912 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.857393 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.857813 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.859078 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.861169 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.862449 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.863444 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.863629 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.868095 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.868107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.868121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.868162 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.871367 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.872025 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.872031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.872037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.872228 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.872686 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.872690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.872692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.873172 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.873398 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.873880 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.874229 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":298,"threads_started":1}
I20260619 02:45:01.880293 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.865866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.7.246.126:36239)
0619 02:45:01.866062 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.866068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.866573 (+ 505us) server_negotiation.cc:407] Connection header received
0619 02:45:01.866732 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.866738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.866828 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.866936 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.868708 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.869862 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.871263 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.871637 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.871771 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.875110 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.875116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.875120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.875156 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.877926 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.878677 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.878680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.878682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.878806 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.879201 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.879204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.879206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.879447 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.879559 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.879919 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.880076 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:45:01.880375 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.865729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43980)
0619 02:45:01.866423 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.866462 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.866616 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.866992 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.866997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.868162 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.868512 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.868523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.869989 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.869995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.871118 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.871131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.871804 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.871811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.871988 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.872576 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.872598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.874936 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.878104 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.878114 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.878120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.878577 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.878927 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.878931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.878934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.879101 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.879550 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.879555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.879691 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.879844 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.880082 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":343,"threads_started":1}
I20260619 02:45:01.881681 15015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.124@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
I20260619 02:45:01.893692 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.882268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45040)
0619 02:45:01.882467 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.882486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.882581 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.882849 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.882852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.883145 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.883268 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.883275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.884580 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.884584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.885512 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.885521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.886160 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.886164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.886295 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.886815 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.886830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.888708 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.891589 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.891593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.891596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.891956 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.892268 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.892271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.892273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.892398 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.892849 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.892852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.892993 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.893108 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.893315 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:45:01.893780 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.882466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.7.246.125:45621)
0619 02:45:01.882625 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.882629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.882643 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:45:01.882689 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.882692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.882745 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.882821 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.883384 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.884466 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.885660 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.886035 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.886121 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.888879 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.888884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.888887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.888917 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.891474 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.892056 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.892060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.892062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.892175 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.892492 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.892495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.892497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.892750 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.892854 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.893190 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.893385 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:45:01.894622 15015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837101, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.124@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
I20260619 02:45:01.906278 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.895273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45278)
0619 02:45:01.895485 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.895510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.895595 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.895830 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.895833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.896133 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.896262 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.896269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.897135 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.897138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.898021 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.898031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.898682 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.898686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.898825 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.899328 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.899344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.901387 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.904201 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.904204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.904207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.904539 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.904847 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.904850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.904852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.904981 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.905421 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.905424 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.905666 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.905778 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.905967 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 02:45:01.906416 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.895403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.7.246.124:44117)
0619 02:45:01.895554 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.895558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.895573 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:45:01.895678 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.895682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.895729 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.895800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.896366 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.897037 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.898145 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.898556 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.898647 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.901615 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.901627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.901631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.901659 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.904063 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.904635 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.904638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.904640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.904757 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.905071 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.905074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.905076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.905320 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.905445 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.905862 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.906086 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":58}
I20260619 02:45:01.911777 15015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Bootstrap starting.
I20260619 02:45:01.914887 15015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:01.915935 15015 log.cc:824] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:01.918570 15015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: No bootstrap required, opened a new log
I20260619 02:45:01.925868 15015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } }
I20260619 02:45:01.926190 15015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:01.926270 15015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22aeb734eae949f1aaee5d2ae407820a, State: Initialized, Role: FOLLOWER
I20260619 02:45:01.926813 15015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } }
I20260619 02:45:01.927599 15021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } } }
I20260619 02:45:01.927934 15021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:01.928740 15015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:45:01.937938 15032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:45:01.938125 15032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:45:01.942128 14861 master_runner.cc:428] Detected that this master 22aeb734eae949f1aaee5d2ae407820a is joining an existing cluster
I20260619 02:45:01.942257 14861 master_runner.cc:432] Initiating AddMaster RPC to add 127.7.246.124:44117
I20260619 02:45:01.966357 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.946721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.7.246.126:36239)
0619 02:45:01.946917 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.946924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.947547 (+ 623us) server_negotiation.cc:407] Connection header received
0619 02:45:01.947832 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.947840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.947943 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.948049 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.949894 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.951468 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.954302 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.955246 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.955382 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.958891 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.958901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.958909 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.958961 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.963113 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.963926 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.963933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.963939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.964094 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.964542 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.964547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.964551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.964860 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.964991 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.965765 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.965985 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:45:01.968225 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.946779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43988)
0619 02:45:01.947443 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.947465 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.947666 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.948316 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.948323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.949192 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.949682 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.949699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.951733 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.951741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.954083 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.954102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.955766 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.955775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.956076 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.957030 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.957063 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.958627 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.963261 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.963267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.963273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.963804 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.964234 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.964238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.964242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.964416 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.965085 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.965089 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.965370 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.967306 (+ 1936us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.967622 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":22,"thread_start_us":375,"threads_started":1}
I20260619 02:45:01.972574 14948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.947879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.7.246.125:45621)
0619 02:45:01.948056 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.948062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.954954 (+ 6892us) server_negotiation.cc:407] Connection header received
0619 02:45:01.955451 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.955457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.955540 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.955612 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.957081 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.958179 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.960794 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.961441 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.961563 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.964183 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.964190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.964196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.964234 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.968395 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.969225 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.969231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.969234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.969409 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.969925 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.969930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.969934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.970256 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.970391 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.971976 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.972212 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:45:01.973608 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.947646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45048)
0619 02:45:01.954842 (+ 7196us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.954863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.955264 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.955827 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.955832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.956536 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.956888 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.956901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.958392 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.958398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.960599 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.960614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.961844 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.961850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.962021 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.962652 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.962677 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.963992 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.968562 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.968567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.968571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.969039 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.969588 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.969592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.969596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.969771 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.970479 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.970487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.971691 (+ 1204us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.972791 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.973116 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7038,"thread_start_us":14199,"threads_started":1}
I20260619 02:45:01.978037 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.947106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.7.246.124:44117)
0619 02:45:01.947304 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.947310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.961874 (+ 14564us) server_negotiation.cc:407] Connection header received
0619 02:45:01.962236 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.962243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.962309 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.962373 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.963854 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.964908 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.967282 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.967970 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.968056 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.970392 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.970398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.970401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.970430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.975161 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.975747 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.975752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.975755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.976123 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.976514 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.976522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.976524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.976771 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.976937 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.977281 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.977635 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":57}
I20260619 02:45:01.978408 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.947655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45280)
0619 02:45:01.961774 (+ 14119us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.961795 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.962106 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.962584 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.962589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.963340 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.963714 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.963728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.965086 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.965090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.967133 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.967147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.968338 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.968346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.968489 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.969055 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.969117 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.970219 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.975303 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.975306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.975309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.975632 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.976246 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.976252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.976256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.976376 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.976887 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.976894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.977035 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.977196 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.977480 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13962,"mutex_wait_us":420,"thread_start_us":6895,"threads_started":1}
I20260619 02:45:01.981264 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14360
Jun 19 02:45:01 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837099, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.126@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
I20260619 02:45:01.995890 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.974308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45286)
0619 02:45:01.974642 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:01.974661 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:01.974766 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:01.978680 (+ 3914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:01.978686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:01.979037 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:01.979172 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.979181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.980471 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.980478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.981857 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:01.981874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.982929 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.982939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:01.983178 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.983978 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:01.984008 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:01.987538 (+ 3530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:01.993138 (+ 5600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.993143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.993147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.993678 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.994117 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:01.994121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:01.994125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:01.994276 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:01.994892 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:01.994897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:01.995081 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 02:45:01.995225 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:45:01.995503 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260619 02:45:01.995901 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.974980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.7.246.124:44117)
0619 02:45:01.978257 (+ 3277us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.978262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.978278 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:01.978368 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.978376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.978512 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.979288 (+ 776us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.979417 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.980324 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.982031 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.982712 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.987774 (+ 5062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:01.988025 (+ 251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:01.988034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:01.988042 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:01.988082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:01.992971 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.993838 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.993843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.993847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.993989 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:01.994414 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:01.994419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:01.994423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:01.994766 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:01.994915 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:01.995325 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 02:45:01.995547 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3185,"thread_start_us":16207,"threads_started":1}
I20260619 02:45:01.997293 14299 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.7.246.124:44117: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a"
member_type: NON_VOTER
last_known_addr {
host: "127.7.246.124"
port: 44117
}
attrs {
promote: true
}
}
dest_uuid: "246a54d557974c13a38a4faee724048b"
cas_config_opid_index: 5
I20260619 02:45:01.998353 14299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } }
I20260619 02:45:02.000125 14859 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.65@KRBTEST.COM
I20260619 02:45:02.004323 14859 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.7.246.124:44117
I20260619 02:45:02.005292 14908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Refusing update from remote peer 246a54d557974c13a38a4faee724048b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:45:02.006292 14861 master.cc:537] Master@127.7.246.124:44117 shutting down...
I20260619 02:45:02.018034 14859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Connected to new peer: Peer: permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:45:02.020771 14908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.024554 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.024848 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:45:02.025211 14859 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: attempting to promote NON_VOTER 550ed9b2244d4d0ebcc10a7c63e7df82 to VOTER
I20260619 02:45:02.025995 14859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } }
I20260619 02:45:02.027318 15043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: NON_VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: true } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.028064 15043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:02.034510 14861 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:02.034997 14861 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:02.035231 14861 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: stopping tablet replica
I20260619 02:45:02.035871 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:01.982584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.7.246.65:39293)
0619 02:45:01.991359 (+ 8775us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:01.991366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:01.991391 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:45:01.991448 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:01.991454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:01.991678 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:01.991868 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:01.993688 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.994909 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.996711 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:01.997388 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:01.997712 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.003671 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.003692 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.003717 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.003779 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.021920 (+ 18141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.031314 (+ 9394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.031328 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.031334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.031557 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.032270 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.032277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.032283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.032829 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.034669 (+ 1840us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.034880 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.035245 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8597,"thread_start_us":8449,"threads_started":1}
I20260619 02:45:02.037575 14483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:02.042055 14908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 LEARNER]: Refusing update from remote peer 246a54d557974c13a38a4faee724048b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 02:45:02.042690 15044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Connected to new peer: Peer: permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 02:45:02.045948 14908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.050046 14936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.050282 14936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:02.054840 15044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.058876 14861 master.cc:552] Master@127.7.246.124:44117 shutdown complete.
I20260619 02:45:02.059024 14861 master_runner.cc:305] Clearing existing system tablet
I20260619 02:45:02.059569 14861 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:02.059814 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260619 02:45:02.065878 14286 connection.cc:698] client connection to 127.7.246.124:44117 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260619 02:45:02.068039 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.067589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45298)
0619 02:45:02.067743 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.067871 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:44117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 02:45:02.068361 14286 consensus_peers.cc:487] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b -> Peer 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124:44117): Couldn't send request to peer 22aeb734eae949f1aaee5d2ae407820a. Status: Network error: Client connection negotiation failed: client connection to 127.7.246.124:44117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:45:02.074509 14861 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.005s sys 0.000s
I20260619 02:45:02.077045 15044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.077425 15044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:02.078255 15050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.078685 15050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:02.079192 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:02.079798 14861 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:02.080075 14861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:45:02.080178 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "22aeb734eae949f1aaee5d2ae407820a"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "d33cb640799cc2d3e1f5d1bc7a93406c"
server_key_iv: "86dd8c3b950412db09de631879413992"
server_key_version: "encryptionkey@0"
I20260619 02:45:02.083570 14861 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:02.088195 14861 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:45:02.088359 14861 log.cc:1196] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:45:02.088765 14861 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Deleting consensus metadata
I20260619 02:45:02.089010 14861 master_runner.cc:315] Copying system tablet from 127.7.246.126:36239
I20260619 02:45:02.091491 14861 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.246.126:36239
I20260619 02:45:02.109361 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.092242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.7.246.126:36239)
0619 02:45:02.092547 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.092556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.094524 (+ 1968us) server_negotiation.cc:407] Connection header received
0619 02:45:02.094745 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.094751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.094825 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.094935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:02.095917 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.097053 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.098924 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.099566 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.099652 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.102050 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.102057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.102062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.102097 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.106004 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.106778 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.106783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.106787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.106938 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.107345 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.107350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.107354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.107699 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.107821 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.108782 (+ 961us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.109006 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 02:45:02.110416 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.092221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:43992)
0619 02:45:02.094410 (+ 2189us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.094429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:02.094587 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:02.094964 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:02.094969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:02.095510 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:02.095750 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.095761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.097237 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.097242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.098757 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.098771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.099908 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.099913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.100097 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.100809 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:02.100831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:02.101853 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:02.106145 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.106151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.106160 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.106652 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.107070 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.107075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.107079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.107231 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.107896 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:02.107901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:02.108488 (+ 587us) client_negotiation.cc:769] Sending connection context
0619 02:45:02.109632 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0619 02:45:02.109971 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":1933,"threads_started":1}
I20260619 02:45:02.110961 14319 tablet_copy_service.cc:140] P 246a54d557974c13a38a4faee724048b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 22aeb734eae949f1aaee5d2ae407820a ({username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:43992)
I20260619 02:45:02.111119 14319 tablet_copy_service.cc:161] P 246a54d557974c13a38a4faee724048b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 22aeb734eae949f1aaee5d2ae407820a at {username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:43992: session id = 22aeb734eae949f1aaee5d2ae407820a-00000000000000000000000000000000
I20260619 02:45:02.114801 14319 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:45:02.116621 14861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:45:02.122952 14861 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: tablet copy: Starting download of 0 data blocks...
I20260619 02:45:02.123253 14861 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: tablet copy: Starting download of 1 WAL segments...
I20260619 02:45:02.125464 14861 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:45:02.128556 14319 tablet_copy_service.cc:342] P 246a54d557974c13a38a4faee724048b: Request end of tablet copy session 22aeb734eae949f1aaee5d2ae407820a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:43992
I20260619 02:45:02.128690 14319 tablet_copy_service.cc:433] P 246a54d557974c13a38a4faee724048b: ending tablet copy session 22aeb734eae949f1aaee5d2ae407820a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 22aeb734eae949f1aaee5d2ae407820a
I20260619 02:45:02.138093 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:02.140614 15060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:02.141973 15061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:02.149966 14861 server_base.cc:1034] running on GCE node
W20260619 02:45:02.150338 15063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:02.150753 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:02.151458 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:02.152648 14861 hybrid_clock.cc:648] HybridClock initialized: now 1781837102152613 us; error 29 us; skew 500 ppm
I20260619 02:45:02.152882 14861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837102, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:02.154820 14861 init.cc:377] Logged in from keytab as oryx/127.7.246.124@KRBTEST.COM (short username oryx)
I20260619 02:45:02.158457 14861 webserver.cc:460] Webserver started at http://127.7.246.124:44161/ using document root <none> and password file <none>
I20260619 02:45:02.158746 14861 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:02.158864 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:02.174161 14861 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.008s
I20260619 02:45:02.175941 15070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:02.176410 14861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:45:02.176608 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "22aeb734eae949f1aaee5d2ae407820a"
format_stamp: "Formatted at 2026-06-19 02:45:01 on dist-test-slave-9tkz"
server_key: "d33cb640799cc2d3e1f5d1bc7a93406c"
server_key_iv: "86dd8c3b950412db09de631879413992"
server_key_version: "encryptionkey@0"
I20260619 02:45:02.176868 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:02.208256 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14360
I20260619 02:45:02.209904 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:02.213585 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:02.233582 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:39293
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:36239
--webserver_port=41747
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:45:02.240041 14861 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.124:44117
I20260619 02:45:02.240063 15121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.124:44117 every 8 connection(s)
I20260619 02:45:02.241178 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
I20260619 02:45:02.246723 15123 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:45:02.250484 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Bootstrap starting.
I20260619 02:45:02.269826 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:45:02.270335 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Bootstrap complete.
I20260619 02:45:02.271354 15123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } }
I20260619 02:45:02.271636 15123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22aeb734eae949f1aaee5d2ae407820a, State: Initialized, Role: LEARNER
I20260619 02:45:02.271884 15123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } }
I20260619 02:45:02.273988 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.274300 15123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:02.274421 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: LEARNER
I20260619 02:45:02.291689 15137 catalog_manager.cc:1259] Loaded cluster ID: 61838aebcd5246fe86c6a4b05f5c3e0e
I20260619 02:45:02.291800 15137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: loading cluster ID for follower catalog manager: success
I20260619 02:45:02.295204 15137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: acquiring CA information for follower catalog manager: success
I20260619 02:45:02.297580 15137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:45:02.589527 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:02.589926 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:02.590082 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:02.598210 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:02.598376 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:02.598552 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:02.608333 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.601176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45300)
0619 02:45:02.602076 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.602093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:02.602217 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:02.602599 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:02.602602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:02.602621 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:02.602761 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.602768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.604902 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.604906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.606129 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.606142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.607680 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.607684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.607902 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.607927 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:45:02.608006 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:45:02.608102 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":703,"threads_started":1}
I20260619 02:45:02.608508 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.601325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.7.246.124:44117)
0619 02:45:02.601951 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.601958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.602182 (+ 224us) server_negotiation.cc:407] Connection header received
0619 02:45:02.602339 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.602345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.602439 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.602581 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:02.602866 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.604738 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.606290 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.607517 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.607651 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.607894 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.608048 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.608183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":331,"threads_started":1}
I20260619 02:45:02.609524 15097 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 02:45:02.610247 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246a54d557974c13a38a4faee724048b. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:02.610563 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: LEARNER
I20260619 02:45:02.611971 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:39293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=41747
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:02.613222 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:02.614580 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:02.628597 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:02.628676 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:02.628703 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:02.629441 15122 server_base.cc:1034] running on GCE node
I20260619 02:45:02.630076 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:02.631083 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:02.632308 15122 hybrid_clock.cc:648] HybridClock initialized: now 1781837102632257 us; error 35 us; skew 500 ppm
I20260619 02:45:02.632692 15122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837102, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:02.636128 15122 init.cc:377] Logged in from keytab as oryx/127.7.246.65@KRBTEST.COM (short username oryx)
I20260619 02:45:02.637755 15122 webserver.cc:460] Webserver started at http://127.7.246.65:41747/ using document root <none> and password file <none>
I20260619 02:45:02.638485 15122 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:02.638643 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:02.644248 15122 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:45:02.647329 15154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:02.648732 15122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:45:02.648947 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "e7478f303cc94ce6af89c965b493a3a0"
format_stamp: "Formatted at 2026-06-19 02:44:59 on dist-test-slave-9tkz"
server_key: "4a5101e6a94efd5232ad9274c523a6d4"
server_key_iv: "e9c0e5cb33e150e6e4ad8502adc4de03"
server_key_version: "encryptionkey@0"
I20260619 02:45:02.650602 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:02.674572 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:02.679687 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:02.679979 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:02.681140 15122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:02.682790 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:02.682943 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:02.683475 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:02.683588 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:02.722330 15122 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:39293
I20260619 02:45:02.722434 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:39293 every 8 connection(s)
I20260619 02:45:02.723932 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:02.733217 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15122
I20260619 02:45:02.733649 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14494
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837102, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.66@KRBTEST.COM
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837102, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
Jun 19 02:45:02 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837102, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
I20260619 02:45:02.762161 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.727058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52993 (local address 127.7.246.124:44117)
0619 02:45:02.727687 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.727698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.732242 (+ 4544us) server_negotiation.cc:407] Connection header received
0619 02:45:02.737892 (+ 5650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.737898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.737998 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.738102 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:02.741275 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.742591 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.745865 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.746624 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.746769 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.752815 (+ 6046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.752822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.752827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.752867 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.757454 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.758779 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.758785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.758789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.758951 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.759634 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.759639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.759643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.760382 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.760533 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.761571 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.761801 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260619 02:45:02.764003 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.727606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.7.246.65:52993)
0619 02:45:02.731954 (+ 4348us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.732031 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:02.737467 (+ 5436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:02.738809 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:02.738827 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:02.739859 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:02.740999 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.741030 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.742863 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.742874 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.745650 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.745673 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.747253 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.747260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.747614 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.748532 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:02.748564 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:02.752571 (+ 4007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:02.757660 (+ 5089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.757667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.757682 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.758595 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.759117 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.759121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.759126 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.759424 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.760520 (+ 1096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:02.760528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:02.761083 (+ 555us) client_negotiation.cc:769] Sending connection context
0619 02:45:02.762374 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0619 02:45:02.762782 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3849,"thread_start_us":21747,"threads_started":1}
I20260619 02:45:02.765714 15267 heartbeater.cc:347] Connected to a master server at 127.7.246.124:44117
I20260619 02:45:02.765982 15267 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:02.766883 15267 heartbeater.cc:511] Master 127.7.246.124:44117 requested a full tablet report, sending...
I20260619 02:45:02.768730 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.732863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:46159 (local address 127.7.246.126:36239)
0619 02:45:02.747989 (+ 15126us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.748008 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.749875 (+ 1867us) server_negotiation.cc:407] Connection header received
0619 02:45:02.750154 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.750160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.750241 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.751409 (+ 1168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:02.752689 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.753834 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.756091 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.756734 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.756830 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.761568 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.761575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.761580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.761617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.765352 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.766210 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.766215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.766218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.766356 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.766819 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.766824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.766826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.767182 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.767325 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.768065 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.768279 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14950,"thread_start_us":15148,"threads_started":1}
I20260619 02:45:02.769651 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.739497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:46159)
0619 02:45:02.749773 (+ 10276us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.749795 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:02.749977 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:02.751566 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:02.751573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:02.752166 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:02.752523 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.752534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.754089 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.754094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.755904 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.755916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.757144 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.757151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.757365 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.757998 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:02.758023 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:02.761338 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:02.765489 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.765493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.765497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.766071 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.766498 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.766503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.766509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.766680 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.767396 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:02.767401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:02.767573 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 02:45:02.767707 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:45:02.768643 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10148,"thread_start_us":1327,"threads_started":1}
I20260619 02:45:02.769711 15087 ts_manager.cc:194] Registered new tserver with Master: e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:45:02.770448 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.727304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:60491 (local address 127.7.246.125:45621)
0619 02:45:02.748645 (+ 21341us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.748654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.748684 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:45:02.748757 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.748765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.748876 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.749025 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:02.751389 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.752925 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.755466 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.756317 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.756460 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.762625 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.762635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.762641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.762688 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.767146 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.768230 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.768235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.768239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.768407 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.768841 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.768846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.768849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.769192 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.769306 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.769845 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.770086 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21178,"thread_start_us":21851,"threads_started":1}
I20260619 02:45:02.770505 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.727606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.7.246.65:60491)
0619 02:45:02.743316 (+ 15710us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:02.743345 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:02.743476 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:02.749815 (+ 6339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:02.749822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:02.750429 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:02.751208 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.751221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.753121 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.753129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.755265 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:02.755280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.756768 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.756775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:02.757094 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.757800 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:02.757828 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:02.762036 (+ 4208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:02.767327 (+ 5291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.767332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.767336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.768069 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.768540 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:02.768545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:02.768549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:02.768682 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:02.769368 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:02.769372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:02.769600 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 02:45:02.769750 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:45:02.770063 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13396,"mutex_wait_us":36,"spinlock_wait_cycles":70528,"thread_start_us":3483,"threads_started":1}
I20260619 02:45:02.772095 15268 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:45:02.772236 15268 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:02.772444 15269 heartbeater.cc:347] Connected to a master server at 127.7.246.125:45621
I20260619 02:45:02.772564 15269 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:02.773397 15268 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:45:02.773595 15269 heartbeater.cc:511] Master 127.7.246.125:45621 requested a full tablet report, sending...
I20260619 02:45:02.774276 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:02.735265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.7.246.66:39321)
0619 02:45:02.737621 (+ 2356us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:02.737631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:02.737664 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:45:02.737749 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:02.737763 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:02.738031 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:02.738229 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:02.743103 (+ 4874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.747375 (+ 4272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.749960 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:02.750925 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:02.751079 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:02.758928 (+ 7849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:02.758947 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:02.758970 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:02.759028 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:02.766813 (+ 7785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.768024 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.768035 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.768038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.768223 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:02.771440 (+ 3217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:02.771446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:02.771450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:02.771913 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:02.772216 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:02.773477 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0619 02:45:02.773815 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2157,"thread_start_us":1943,"threads_started":1}
I20260619 02:45:02.774708 14299 ts_manager.cc:194] Re-registered known tserver with Master: e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:45:02.775389 14898 ts_manager.cc:194] Registered new tserver with Master: e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:45:02.775661 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:46159
I20260619 02:45:02.775772 14617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:02.862896 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14494
I20260619 02:45:02.879971 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:39321
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:36239
--webserver_port=40357
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:45:03.054826 15050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: NON_VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: true } } }
I20260619 02:45:03.055160 15050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:03.056704 15280 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: attempting to promote NON_VOTER 22aeb734eae949f1aaee5d2ae407820a to VOTER
I20260619 02:45:03.057271 15280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } }
I20260619 02:45:03.059152 14908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Refusing update from remote peer 246a54d557974c13a38a4faee724048b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 02:45:03.059185 15097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 LEARNER]: Refusing update from remote peer 246a54d557974c13a38a4faee724048b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 02:45:03.059835 15281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Connected to new peer: Peer: permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:45:03.060329 15050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:45:03.065057 14908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.064851 15097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.064143 15280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.069137 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.069476 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:03.073463 15050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.073865 15050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:03.073923 15285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.074266 15285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:03.074232 15050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.074549 15050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:03.074890 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "246a54d557974c13a38a4faee724048b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:03.075110 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b [sys.catalog]: This master's current role is: LEADER
W20260619 02:45:03.225760 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:03.226040 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:03.226135 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:03.233376 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:03.233528 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:03.233727 15279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:03.247997 15279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:39321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=40357
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:03.249670 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:03.251304 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:03.264922 15290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:03.264830 15293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:03.264962 15291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:03.266286 15279 server_base.cc:1034] running on GCE node
I20260619 02:45:03.266909 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:03.267840 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:03.269060 15279 hybrid_clock.cc:648] HybridClock initialized: now 1781837103268983 us; error 65 us; skew 500 ppm
I20260619 02:45:03.269430 15279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:03.272804 15279 init.cc:377] Logged in from keytab as oryx/127.7.246.66@KRBTEST.COM (short username oryx)
I20260619 02:45:03.274677 15279 webserver.cc:460] Webserver started at http://127.7.246.66:40357/ using document root <none> and password file <none>
I20260619 02:45:03.275503 15279 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:03.275660 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:03.281253 15279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:45:03.284101 15300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.285494 15279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:45:03.285697 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "ba3f228c8d1c425184bc717e23396549"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "f37232643c8abe93e1ee6b0f85ec7aec"
server_key_iv: "7fe0d37e5797e06ea062b5560a067b69"
server_key_version: "encryptionkey@0"
I20260619 02:45:03.286755 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:03.300865 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:03.305123 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:03.305424 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:03.306424 15279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:03.307850 15279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:03.307967 15279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.308106 15279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:03.308193 15279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.342679 15279 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:39321
I20260619 02:45:03.342730 15412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:39321 every 8 connection(s)
I20260619 02:45:03.344287 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:03.351905 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15279
I20260619 02:45:03.352246 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14627
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.67@KRBTEST.COM
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
I20260619 02:45:03.392678 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.347230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:46127 (local address 127.7.246.126:36239)
0619 02:45:03.354597 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.354607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.361722 (+ 7115us) server_negotiation.cc:407] Connection header received
0619 02:45:03.366471 (+ 4749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:03.366480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:03.366600 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:03.366763 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:03.369835 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.371354 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.375096 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.376041 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.376185 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.384702 (+ 8517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:03.384709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:03.384714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:03.384757 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:03.388673 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.389702 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.389711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.389717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.389871 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.390413 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.390418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.390420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.390757 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:03.390910 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:03.392051 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0619 02:45:03.392291 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7199,"thread_start_us":8842,"threads_started":1}
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
I20260619 02:45:03.394269 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.353519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.7.246.67:46829)
0619 02:45:03.366004 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.366015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.367722 (+ 1707us) server_negotiation.cc:407] Connection header received
0619 02:45:03.367831 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:03.367856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:03.368165 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:03.371163 (+ 2998us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:03.371484 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.373590 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.376194 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.377178 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.381854 (+ 4676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.385902 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:03.385921 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:03.385952 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:03.385994 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:03.390737 (+ 4743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.391621 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.391631 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.391634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.391787 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.392259 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.392264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.392267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.392674 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:03.392978 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:03.393592 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 02:45:03.393854 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12227,"thread_start_us":12471,"threads_started":1}
I20260619 02:45:03.395030 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.348131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:46127)
0619 02:45:03.361452 (+ 13321us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:03.361561 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:03.366044 (+ 4483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:03.367369 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:03.367387 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:03.368435 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:03.369532 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.369602 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.371618 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.371625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.374884 (+ 3259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.374898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.376627 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.376636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.377011 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.378210 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:03.378258 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:03.384394 (+ 6136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:03.388835 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.388845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.388864 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.389529 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.390025 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.390029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.390033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.390268 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.391047 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:03.391055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:03.391569 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 02:45:03.392977 (+ 1408us) client_negotiation.cc:240] Negotiation successful
0619 02:45:03.393575 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12840,"thread_start_us":25837,"threads_started":1}
I20260619 02:45:03.395524 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.348131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.7.246.66:47099)
0619 02:45:03.374535 (+ 26404us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:03.374555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:03.374684 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:03.375246 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:03.375251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:03.375900 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:03.376192 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.376203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.377947 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.377955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.379917 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.379935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.381250 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.381256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.381463 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.382157 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:03.382189 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:03.386125 (+ 3936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:03.391406 (+ 5281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.391411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.391414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.392152 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.392645 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.392651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.392658 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.393788 (+ 1130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.394517 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:03.394523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:03.394674 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 02:45:03.394814 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:45:03.395095 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26235,"mutex_wait_us":40,"spinlock_wait_cycles":70400,"thread_start_us":13413,"threads_started":1}
I20260619 02:45:03.395744 14750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:03.396301 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.347274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:47099 (local address 127.7.246.124:44117)
0619 02:45:03.374198 (+ 26924us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.374205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.374784 (+ 579us) server_negotiation.cc:407] Connection header received
0619 02:45:03.374846 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:03.374851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:03.374926 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:03.375022 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:03.376369 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.377412 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.380104 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.380850 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.380974 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.386342 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:03.386359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:03.386365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:03.386404 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:03.390331 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.392292 (+ 1961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.392298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.392302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.392496 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.393940 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.393946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.393949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.394290 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:03.394414 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:03.394976 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 02:45:03.395508 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26718,"thread_start_us":26609,"threads_started":1}
I20260619 02:45:03.397109 15414 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:45:03.397392 15414 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:03.397830 15413 heartbeater.cc:347] Connected to a master server at 127.7.246.124:44117
I20260619 02:45:03.398079 15413 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:03.398443 15414 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:45:03.398648 15413 heartbeater.cc:511] Master 127.7.246.124:44117 requested a full tablet report, sending...
I20260619 02:45:03.400157 14299 ts_manager.cc:194] Re-registered known tserver with Master: ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321)
I20260619 02:45:03.400568 15087 ts_manager.cc:194] Registered new tserver with Master: ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321)
I20260619 02:45:03.401763 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:46127
I20260619 02:45:03.403427 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.377778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.7.246.66:49323)
0619 02:45:03.384008 (+ 6230us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:03.384030 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:03.384146 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:03.385282 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:03.385289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:03.386065 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:03.386380 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.386394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.388027 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.388033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.389644 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.389659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.390611 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.390620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.390802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.391414 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:03.391442 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:03.396083 (+ 4641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:03.400112 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.400119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.400126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.401029 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.401478 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:03.401483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:03.401487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:03.401679 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:03.402378 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:03.402388 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:03.402542 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 02:45:03.402717 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 02:45:03.403014 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6065,"thread_start_us":5885,"threads_started":1}
I20260619 02:45:03.404250 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.374857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:49323 (local address 127.7.246.125:45621)
0619 02:45:03.384926 (+ 10069us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.384933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.384953 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:45:03.385011 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:03.385016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:03.385094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:03.385217 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:03.386558 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.387836 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.389823 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.390444 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.390565 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.396296 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:03.396305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:03.396310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:03.396386 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:03.399957 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.401184 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.401190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.401194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.401336 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:03.401813 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:03.401818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:03.401822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:03.402157 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:03.402308 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:03.403700 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0619 02:45:03.403916 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9930,"thread_start_us":9802,"threads_started":1}
I20260619 02:45:03.405280 15415 heartbeater.cc:347] Connected to a master server at 127.7.246.125:45621
I20260619 02:45:03.405440 15415 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:03.405900 15415 heartbeater.cc:511] Master 127.7.246.125:45621 requested a full tablet report, sending...
I20260619 02:45:03.407047 14898 ts_manager.cc:194] Registered new tserver with Master: ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321)
I20260619 02:45:03.488502 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14627
I20260619 02:45:03.504709 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:46829
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:36239
--webserver_port=33037
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:45:03.778203 15268 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
W20260619 02:45:03.833477 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:03.833838 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:03.833935 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:03.841879 15426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:03.841995 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:03.842110 15426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:03.852298 15426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:46829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=33037
--tserver_master_addrs=127.7.246.126:36239,127.7.246.125:45621,127.7.246.124:44117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:03.853679 15426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:03.855101 15426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:03.868427 15434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:03.868464 15432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:03.868736 15431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:03.870755 15426 server_base.cc:1034] running on GCE node
I20260619 02:45:03.871320 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:03.872238 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:03.873431 15426 hybrid_clock.cc:648] HybridClock initialized: now 1781837103873363 us; error 62 us; skew 500 ppm
I20260619 02:45:03.873776 15426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:03.876930 15426 init.cc:377] Logged in from keytab as oryx/127.7.246.67@KRBTEST.COM (short username oryx)
I20260619 02:45:03.878729 15426 webserver.cc:460] Webserver started at http://127.7.246.67:33037/ using document root <none> and password file <none>
I20260619 02:45:03.879541 15426 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:03.879693 15426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:03.885195 15426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:03.888207 15441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.889741 15426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:03.889940 15426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "ad410b9ad6704ad095daa5fbbcf3088c"
format_stamp: "Formatted at 2026-06-19 02:45:00 on dist-test-slave-9tkz"
server_key: "351f46bb1a26c5565bad7c8b5499a5af"
server_key_iv: "e5fa4ea8bd88c193da4070ebd671c80d"
server_key_version: "encryptionkey@0"
I20260619 02:45:03.891083 15426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:03.911521 15426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:03.916261 15426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:03.916555 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:03.917783 15426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:03.919339 15426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:03.919477 15426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.919605 15426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:03.919692 15426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:03.952925 15426 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:46829
I20260619 02:45:03.952967 15553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:46829 every 8 connection(s)
I20260619 02:45:03.954524 15426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:03.955629 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15426
Jun 19 02:45:03 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:04.017620 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.957827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:52909 (local address 127.7.246.126:36239)
0619 02:45:03.979552 (+ 21725us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.979562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.983647 (+ 4085us) server_negotiation.cc:407] Connection header received
0619 02:45:03.984259 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:03.984270 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:03.984421 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:03.984583 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:03.987833 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.989561 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.993804 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.994864 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.995041 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.007048 (+ 12007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.007060 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.007068 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.007129 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.012654 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.013968 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.013977 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.013983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.014168 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.014800 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.014807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.014813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.015209 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.016735 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.016894 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.017176 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21378,"thread_start_us":21480,"threads_started":1}
I20260619 02:45:04.021785 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.959252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:52909)
0619 02:45:03.983479 (+ 24227us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:03.983509 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:03.983727 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:03.985040 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:03.985061 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:03.986413 (+ 1352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:03.987561 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.987611 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.989930 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.989940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.993426 (+ 3486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:03.993454 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:03.995665 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:03.995677 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:03.996143 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:03.997360 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:03.997410 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:04.006745 (+ 9335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:04.012877 (+ 6132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.012891 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.012919 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.013770 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.014308 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.014316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.014323 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.014629 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.015372 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:04.015384 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:04.015933 (+ 549us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.016318 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.019710 (+ 3392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24032,"mutex_wait_us":2565,"thread_start_us":25494,"threads_started":1}
I20260619 02:45:04.025471 15555 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36239
I20260619 02:45:04.025811 15555 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:04.026952 15555 heartbeater.cc:511] Master 127.7.246.126:36239 requested a full tablet report, sending...
I20260619 02:45:04.032837 14299 ts_manager.cc:194] Re-registered known tserver with Master: ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:04.034526 14299 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:52909
Jun 19 02:45:04 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.124@KRBTEST.COM
Jun 19 02:45:04 dist-test-slave-9tkz krb5kdc[14254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837103, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.125@KRBTEST.COM
I20260619 02:45:04.056104 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.958527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.7.246.67:40409)
0619 02:45:03.966832 (+ 8305us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:03.966903 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:03.972997 (+ 6094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.004699 (+ 31702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.004708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.005434 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:04.005831 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.005844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.015978 (+ 10134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.015989 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.019020 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.019043 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.042967 (+ 23924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.042977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.043240 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.044149 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:04.044182 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:04.047270 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:04.052935 (+ 5665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.052942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.052946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.053592 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.054063 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.054069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.054073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.054240 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.054946 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:04.054953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:04.055128 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.055286 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.055601 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7804,"thread_start_us":7502,"threads_started":1}
I20260619 02:45:04.057787 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.957819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:40409 (local address 127.7.246.124:44117)
0619 02:45:03.999920 (+ 42101us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:03.999928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:03.999949 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:45:04.000021 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.000027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.000107 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.006078 (+ 5971us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.006247 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.014667 (+ 8420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.040188 (+ 25521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.042685 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.042862 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.047525 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.047534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.047543 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.047591 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.052746 (+ 5155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.053763 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.053769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.053773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.053933 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.054410 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.054416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.054420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.054792 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.056519 (+ 1727us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.056666 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.056901 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41941,"thread_start_us":46449,"threads_started":1}
I20260619 02:45:04.058734 15554 heartbeater.cc:347] Connected to a master server at 127.7.246.124:44117
I20260619 02:45:04.058897 15554 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:04.059355 15554 heartbeater.cc:511] Master 127.7.246.124:44117 requested a full tablet report, sending...
I20260619 02:45:04.060626 15087 ts_manager.cc:194] Registered new tserver with Master: ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:04.063555 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.001379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.7.246.67:58767)
0619 02:45:04.022746 (+ 21367us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.022772 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.022940 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.024489 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.024498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.031165 (+ 6667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:04.031457 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.031476 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.038643 (+ 7167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.038653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.044611 (+ 5958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.044642 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.045993 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.046000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.046222 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.049158 (+ 2936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:04.049186 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:04.053426 (+ 4240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:04.060501 (+ 7075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.060507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.060510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.061100 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.061666 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:04.061671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:04.061674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:04.061833 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:04.062544 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:04.062549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:04.062691 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.062822 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.063166 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21196,"thread_start_us":21671,"threads_started":1}
I20260619 02:45:04.064406 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:03.998377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.7.246.126:36239)
0619 02:45:04.010886 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.010895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.029667 (+ 18772us) server_negotiation.cc:407] Connection header received
0619 02:45:04.037450 (+ 7783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.037470 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.037583 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.037678 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.047633 (+ 9955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.049282 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.051394 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.052039 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.052132 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.056734 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.056740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.056746 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.056790 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.061043 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.061825 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.061830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.061833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.061990 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.062415 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.062420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.062423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.062752 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.062968 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.063537 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.064040 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12326,"thread_start_us":12522,"threads_started":1}
I20260619 02:45:04.064517 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.001505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:58767 (local address 127.7.246.125:45621)
0619 02:45:04.020839 (+ 19334us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.020849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.023197 (+ 2348us) server_negotiation.cc:407] Connection header received
0619 02:45:04.023284 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.023293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.023400 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.023552 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.031634 (+ 8082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.034146 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.044805 (+ 10659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.045482 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.045670 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.053647 (+ 7977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.053657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.053663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.053724 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.059602 (+ 5878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.061281 (+ 1679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.061290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.061295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.061470 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.061988 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.061993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.061996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.062321 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.062472 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.063904 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.064141 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19159,"thread_start_us":19326,"threads_started":1}
I20260619 02:45:04.065210 15556 heartbeater.cc:347] Connected to a master server at 127.7.246.125:45621
I20260619 02:45:04.065389 15556 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:04.065863 15556 heartbeater.cc:511] Master 127.7.246.125:45621 requested a full tablet report, sending...
I20260619 02:45:04.067449 14898 ts_manager.cc:194] Registered new tserver with Master: ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:04.071981 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.030284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.7.246.125:45621)
0619 02:45:04.046933 (+ 16649us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.046940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.046956 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:04.047002 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.047006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.047076 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.047182 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.052985 (+ 5803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.054721 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.057010 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.058174 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.058294 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.063324 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.063332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.063340 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.063391 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.068912 (+ 5521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.069743 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.069749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.069753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.069934 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.070374 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.070379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.070383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.070720 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.070945 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.071370 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.071584 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16521,"thread_start_us":17120,"threads_started":1}
I20260619 02:45:04.074714 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.017527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.7.246.124:44117)
0619 02:45:04.041837 (+ 24310us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.041846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.045410 (+ 3564us) server_negotiation.cc:407] Connection header received
0619 02:45:04.047594 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.047600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.047681 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.047784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.050640 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.052005 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.062049 (+ 10044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.062773 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.062881 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.067926 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.067933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.067939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.067980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.071729 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.072490 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.072495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.072500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.072645 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.073032 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.073037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.073040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.073365 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.073602 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.074170 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.074343 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24130,"thread_start_us":23958,"threads_started":1}
I20260619 02:45:04.069908 14299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:04.079185 14299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:04.127538 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.100339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.7.246.66:39321)
0619 02:45:04.101203 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.101213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.118933 (+ 17720us) server_negotiation.cc:407] Connection header received
0619 02:45:04.119054 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.119063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.119361 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.119567 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:04.120116 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.121635 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.125456 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.126485 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.126637 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.126862 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.127101 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.127254 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":595,"threads_started":1}
I20260619 02:45:04.127717 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.099870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39321 (local address 127.0.0.1:34152)
0619 02:45:04.118138 (+ 18268us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.118164 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.118346 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.119583 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.119588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.119621 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:04.119937 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.119946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.121769 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.121774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.125230 (+ 3456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.125247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.126709 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.126715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.126914 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.126947 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.127054 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.127201 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18070,"mutex_wait_us":6,"thread_start_us":19204,"threads_started":1}
I20260619 02:45:04.130491 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.100324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:46829 (local address 127.0.0.1:38148)
0619 02:45:04.119337 (+ 19013us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.119400 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.120182 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.121796 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.121800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.121858 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:04.122933 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.122946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.125051 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.125057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.127558 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.127572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.129703 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.129708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.129884 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.129914 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.130011 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.130134 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18880,"mutex_wait_us":104,"thread_start_us":20309,"threads_started":1}
I20260619 02:45:04.131170 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.101401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.7.246.67:46829)
0619 02:45:04.117104 (+ 15703us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.117117 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.119501 (+ 2384us) server_negotiation.cc:407] Connection header received
0619 02:45:04.120375 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.120383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.120650 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.120805 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:04.123180 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.124843 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.127731 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.129191 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.129298 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.129504 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.130719 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.130871 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15438,"thread_start_us":15723,"threads_started":1}
I20260619 02:45:04.132016 15348 tablet_service.cc:1467] Processing CreateTablet for tablet ed2a50e568dd439484d43099cbf26f41 (DEFAULT_TABLE table=test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:04.133658 15348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2a50e568dd439484d43099cbf26f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:04.134666 15489 tablet_service.cc:1467] Processing CreateTablet for tablet ed2a50e568dd439484d43099cbf26f41 (DEFAULT_TABLE table=test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:04.135848 15489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2a50e568dd439484d43099cbf26f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:04.141023 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.102579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.7.246.65:39293)
0619 02:45:04.118855 (+ 16276us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.118869 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.121070 (+ 2201us) server_negotiation.cc:407] Connection header received
0619 02:45:04.121161 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.121168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.121365 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.121532 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:04.123453 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.124915 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.127967 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.138019 (+ 10052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.138134 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.138277 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.140436 (+ 2159us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.140682 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16101,"thread_start_us":15976,"threads_started":1}
I20260619 02:45:04.141812 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.102251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:39293 (local address 127.0.0.1:35234)
0619 02:45:04.120404 (+ 18153us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.120421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.120548 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.122379 (+ 1831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.122382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.122403 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:04.123232 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.123250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.125075 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.125082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.127569 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.127578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.140022 (+ 12444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.140029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.140214 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.140249 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.141222 (+ 973us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.141375 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18036,"mutex_wait_us":144,"thread_start_us":15527,"threads_started":1}
I20260619 02:45:04.145453 15202 tablet_service.cc:1467] Processing CreateTablet for tablet ed2a50e568dd439484d43099cbf26f41 (DEFAULT_TABLE table=test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:04.146865 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2a50e568dd439484d43099cbf26f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:04.149052 15592 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Bootstrap starting.
I20260619 02:45:04.152269 15592 tablet_bootstrap.cc:654] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:04.153235 15592 log.cc:824] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:04.156890 15592 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: No bootstrap required, opened a new log
I20260619 02:45:04.157181 15592 ts_tablet_manager.cc:1397] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:45:04.164927 15592 raft_consensus.cc:348] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.165315 15592 raft_consensus.cc:374] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:04.165405 15592 raft_consensus.cc:729] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad410b9ad6704ad095daa5fbbcf3088c, State: Initialized, Role: FOLLOWER
I20260619 02:45:04.166110 15592 consensus_queue.cc:260] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.166628 15591 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Bootstrap starting.
I20260619 02:45:04.169806 15592 ts_tablet_manager.cc:1428] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260619 02:45:04.170090 15555 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
I20260619 02:45:04.171208 15591 tablet_bootstrap.cc:654] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:04.172593 15591 log.cc:824] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:04.173684 15594 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Bootstrap starting.
I20260619 02:45:04.176951 15594 tablet_bootstrap.cc:654] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:04.178319 15594 log.cc:824] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:04.180994 15591 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: No bootstrap required, opened a new log
I20260619 02:45:04.181283 15591 ts_tablet_manager.cc:1397] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260619 02:45:04.181329 15594 tablet_bootstrap.cc:492] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: No bootstrap required, opened a new log
I20260619 02:45:04.181600 15594 ts_tablet_manager.cc:1397] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 02:45:04.192478 15591 raft_consensus.cc:348] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.192482 15594 raft_consensus.cc:348] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.192953 15594 raft_consensus.cc:374] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:04.192963 15591 raft_consensus.cc:374] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:04.193064 15594 raft_consensus.cc:729] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7478f303cc94ce6af89c965b493a3a0, State: Initialized, Role: FOLLOWER
I20260619 02:45:04.193064 15591 raft_consensus.cc:729] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3f228c8d1c425184bc717e23396549, State: Initialized, Role: FOLLOWER
I20260619 02:45:04.193744 15594 consensus_queue.cc:260] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.193746 15591 consensus_queue.cc:260] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.195829 15591 ts_tablet_manager.cc:1428] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Time spent starting tablet: real 0.014s user 0.007s sys 0.009s
I20260619 02:45:04.195829 15594 ts_tablet_manager.cc:1428] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 02:45:04.196125 15414 heartbeater.cc:503] Master 127.7.246.126:36239 was elected leader, sending a full tablet report...
W20260619 02:45:04.234545 15270 tablet.cc:2369] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:45:04.291071 15595 raft_consensus.cc:482] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:04.291390 15595 raft_consensus.cc:504] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.293172 15595 leader_election.cc:288] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321), e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:45:04.299173 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.293755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:46211 (local address 127.7.246.65:39293)
0619 02:45:04.293958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.293962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.293976 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:45:04.294061 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.294066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.294129 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.294226 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:04.294569 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.295376 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.297497 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.298276 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.298428 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.298493 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.298814 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.298934 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:45:04.299201 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.293438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:39293 (local address 127.7.246.67:46211)
0619 02:45:04.293683 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.293697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.293769 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.294261 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.294264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.294284 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:04.294440 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.294449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.295527 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.295531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.297358 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.297372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.298437 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.298443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.298624 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.298671 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.298776 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.298906 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1}
I20260619 02:45:04.299391 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.293599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:59931 (local address 127.7.246.66:39321)
0619 02:45:04.293776 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.293784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.293803 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:45:04.293877 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.293880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.293934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.294011 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:04.294512 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.295349 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.297401 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.298270 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.298440 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.298520 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.299009 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.299130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:45:04.299671 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.293370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39321 (local address 127.7.246.67:59931)
0619 02:45:04.293537 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:04.293577 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:04.293711 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:04.294059 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:04.294063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:04.294091 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:04.294334 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.294348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.295488 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.295495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.297254 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:04.297268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.298615 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.298621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:04.298858 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.298881 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:45:04.298992 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:45:04.299119 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":43}
I20260619 02:45:04.300181 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2a50e568dd439484d43099cbf26f41" candidate_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7478f303cc94ce6af89c965b493a3a0" is_pre_election: true
W20260619 02:45:04.300536 15567 tablet.cc:2369] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:45:04.300609 15368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2a50e568dd439484d43099cbf26f41" candidate_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3f228c8d1c425184bc717e23396549" is_pre_election: true
I20260619 02:45:04.300741 15222 raft_consensus.cc:2454] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad410b9ad6704ad095daa5fbbcf3088c in term 0.
I20260619 02:45:04.301148 15368 raft_consensus.cc:2454] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad410b9ad6704ad095daa5fbbcf3088c in term 0.
I20260619 02:45:04.301455 15445 leader_election.cc:302] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad410b9ad6704ad095daa5fbbcf3088c, e7478f303cc94ce6af89c965b493a3a0; no voters:
I20260619 02:45:04.301884 15595 raft_consensus.cc:2781] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:04.302037 15595 raft_consensus.cc:482] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:04.302134 15595 raft_consensus.cc:3037] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:04.304167 15595 raft_consensus.cc:504] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.304678 15595 leader_election.cc:288] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [CANDIDATE]: Term 1 election: Requested vote from peers ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321), e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293)
I20260619 02:45:04.305059 15368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2a50e568dd439484d43099cbf26f41" candidate_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3f228c8d1c425184bc717e23396549"
I20260619 02:45:04.305107 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2a50e568dd439484d43099cbf26f41" candidate_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7478f303cc94ce6af89c965b493a3a0"
I20260619 02:45:04.305269 15368 raft_consensus.cc:3037] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:04.305275 15222 raft_consensus.cc:3037] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:04.308768 15222 raft_consensus.cc:2454] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad410b9ad6704ad095daa5fbbcf3088c in term 1.
I20260619 02:45:04.308768 15368 raft_consensus.cc:2454] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad410b9ad6704ad095daa5fbbcf3088c in term 1.
I20260619 02:45:04.309154 15442 leader_election.cc:302] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad410b9ad6704ad095daa5fbbcf3088c, ba3f228c8d1c425184bc717e23396549; no voters:
I20260619 02:45:04.309413 15595 raft_consensus.cc:2781] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:04.309866 15595 raft_consensus.cc:686] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 1 LEADER]: Becoming Leader. State: Replica: ad410b9ad6704ad095daa5fbbcf3088c, State: Running, Role: LEADER
I20260619 02:45:04.310477 15595 consensus_queue.cc:237] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } }
I20260619 02:45:04.315918 14298 catalog_manager.cc:5495] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c reported cstate change: term changed from 0 to 1, leader changed from <none> to ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67). New cstate: current_term: 1 leader_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } health_report { overall_health: UNKNOWN } } }
W20260619 02:45:04.347747 15416 tablet.cc:2369] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:45:04.369756 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.364458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.7.246.67:46829)
0619 02:45:04.364648 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.364654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.364798 (+ 144us) server_negotiation.cc:407] Connection header received
0619 02:45:04.365038 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.365041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.365123 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.365279 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:04.365657 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.366329 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.367611 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.368013 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.368144 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.368657 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:04.369154 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:04.369332 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.369399 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.369474 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:45:04.378293 15222 raft_consensus.cc:1261] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Refusing update from remote peer ad410b9ad6704ad095daa5fbbcf3088c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:04.378319 15368 raft_consensus.cc:1261] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Refusing update from remote peer ad410b9ad6704ad095daa5fbbcf3088c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:04.379282 15600 consensus_queue.cc:1035] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:04.379621 15595 consensus_queue.cc:1035] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:04.408198 14299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:04.408519 14299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:04.415350 14299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 246a54d557974c13a38a4faee724048b: Sending DeleteTablet for 3 replicas of tablet ed2a50e568dd439484d43099cbf26f41
I20260619 02:45:04.416455 15348 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2a50e568dd439484d43099cbf26f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:04 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:34152
I20260619 02:45:04.416695 15489 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2a50e568dd439484d43099cbf26f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:04 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38148
I20260619 02:45:04.416778 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2a50e568dd439484d43099cbf26f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:04 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35234
I20260619 02:45:04.428686 15613 tablet_replica.cc:330] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: stopping tablet replica
I20260619 02:45:04.429229 15613 raft_consensus.cc:2229] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:04.430078 15613 raft_consensus.cc:2258] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:04.430516 15614 tablet_replica.cc:330] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: stopping tablet replica
I20260619 02:45:04.431109 15614 raft_consensus.cc:2229] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:04.431793 15614 raft_consensus.cc:2258] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:04.432390 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14267
I20260619 02:45:04.433482 14329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:04.434470 15614 ts_tablet_manager.cc:1905] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:04.443060 15613 ts_tablet_manager.cc:1905] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:04.444942 15614 ts_tablet_manager.cc:1918] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:04.445312 15614 log.cc:1196] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/ed2a50e568dd439484d43099cbf26f41
I20260619 02:45:04.446130 15614 ts_tablet_manager.cc:1939] T ed2a50e568dd439484d43099cbf26f41 P e7478f303cc94ce6af89c965b493a3a0: Deleting consensus metadata
I20260619 02:45:04.450349 15612 tablet_replica.cc:330] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: stopping tablet replica
I20260619 02:45:04.451969 15612 raft_consensus.cc:2229] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:04.452502 15612 raft_consensus.cc:2258] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:04.457408 15613 ts_tablet_manager.cc:1918] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:04.457661 15613 log.cc:1196] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/ed2a50e568dd439484d43099cbf26f41
I20260619 02:45:04.458329 15613 ts_tablet_manager.cc:1939] T ed2a50e568dd439484d43099cbf26f41 P ad410b9ad6704ad095daa5fbbcf3088c: Deleting consensus metadata
I20260619 02:45:04.462771 15612 ts_tablet_manager.cc:1905] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:04.466481 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.423543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.7.246.125:45621)
0619 02:45:04.423788 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.423798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.430988 (+ 7190us) server_negotiation.cc:407] Connection header received
0619 02:45:04.431375 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.431382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.431464 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.431539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.443815 (+ 12276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.445013 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.451304 (+ 6291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.452137 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.452268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.457156 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.457167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.457175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.457236 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.461999 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.463302 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.463311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.463317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.463515 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.464079 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.464087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.464093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.464584 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.464874 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.465666 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.465993 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:45:04.472783 15612 ts_tablet_manager.cc:1918] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:04.472975 15612 log.cc:1196] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/ed2a50e568dd439484d43099cbf26f41
I20260619 02:45:04.473629 15612 ts_tablet_manager.cc:1939] T ed2a50e568dd439484d43099cbf26f41 P ba3f228c8d1c425184bc717e23396549: Deleting consensus metadata
I20260619 02:45:04.485668 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.431163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.7.246.124:44117)
0619 02:45:04.431369 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.431375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.468436 (+ 37061us) server_negotiation.cc:407] Connection header received
0619 02:45:04.468658 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.468665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.468756 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.468891 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.471016 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.472155 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.474372 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.475071 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.475216 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.478559 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.478565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.478568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.478602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.482281 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.483114 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.483120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.483124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.483284 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.483716 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.483723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.483727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.484143 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.484387 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.484993 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.485233 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:45:04.594095 14285 catalog_manager.cc:4841] TS ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829): tablet ed2a50e568dd439484d43099cbf26f41 (table test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]) successfully deleted
W20260619 02:45:04.594537 14285 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.246.67:46829, user_credentials={real_user=slave}} blocked reactor thread for 134295us
I20260619 02:45:04.594729 14286 catalog_manager.cc:4841] TS e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293): tablet ed2a50e568dd439484d43099cbf26f41 (table test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]) successfully deleted
I20260619 02:45:04.594794 14283 catalog_manager.cc:4841] TS ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321): tablet ed2a50e568dd439484d43099cbf26f41 (table test-table [id=70e90bcbe3b4449ca6b12574cd5439f6]) successfully deleted
W20260619 02:45:04.594885 14286 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.246.65:39293, user_credentials={real_user=slave}} blocked reactor thread for 146345us
I20260619 02:45:04.595745 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14267
I20260619 02:45:04.637727 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.620694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.7.246.124:44117)
0619 02:45:04.620929 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.620938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.621901 (+ 963us) server_negotiation.cc:407] Connection header received
0619 02:45:04.622194 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.622201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.622302 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.622377 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.624718 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.625438 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.627135 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.627746 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.627873 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.630668 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.630682 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.630685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.630723 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.634148 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.635006 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.635013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.635018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.635157 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.635499 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.635504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.635507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.635772 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.636001 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.637082 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.637312 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:45:04.637768 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:04.622037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.7.246.125:45621)
0619 02:45:04.622223 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:04.622228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:04.622244 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:04.622302 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:04.622305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:04.622358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:04.622487 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:04.623885 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.624726 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.626631 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:04.627324 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:04.627438 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:04.631106 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:04.631113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:04.631118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:04.631167 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:04.634397 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.635202 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.635208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.635212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.635358 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:04.636309 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:04.636314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:04.636317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:04.636651 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:04.636904 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:04.637170 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 02:45:04.637383 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:45:05.400947 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.400117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:55155)
0619 02:45:05.400582 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.400724 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260619 02:45:05.401093 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.400195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:39501)
0619 02:45:05.400806 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.400925 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":400,"threads_started":1}
W20260619 02:45:05.401409 15555 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:36239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
W20260619 02:45:05.401539 15414 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:36239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.403625 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.402847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:38211)
0619 02:45:05.403292 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.403437 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":235,"threads_started":1}
W20260619 02:45:05.404064 15268 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:36239 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.915293 15626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 246a54d557974c13a38a4faee724048b)
I20260619 02:45:05.915555 15627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 246a54d557974c13a38a4faee724048b)
I20260619 02:45:05.915566 15626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } }
I20260619 02:45:05.915855 15627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } }
I20260619 02:45:05.916366 15626 leader_election.cc:288] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 246a54d557974c13a38a4faee724048b (127.7.246.126:36239), 22aeb734eae949f1aaee5d2ae407820a (127.7.246.124:44117)
I20260619 02:45:05.924080 15627 leader_election.cc:288] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 246a54d557974c13a38a4faee724048b (127.7.246.126:36239), 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125:45621)
I20260619 02:45:05.927810 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.917835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44494)
0619 02:45:05.927271 (+ 9436us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.927493 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9298,"thread_start_us":9102,"threads_started":1}
I20260619 02:45:05.929705 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.916566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44480)
0619 02:45:05.929048 (+ 12482us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.929318 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12285,"threads_started":1}
W20260619 02:45:05.930877 15071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
W20260619 02:45:05.931421 14882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.939074 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.938637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44514)
0619 02:45:05.938777 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.938886 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:45:05.939318 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.935541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44498)
0619 02:45:05.938797 (+ 3256us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.939003 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3117,"threads_started":1}
W20260619 02:45:05.939561 14882 leader_election.cc:334] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 246a54d557974c13a38a4faee724048b (127.7.246.126:36239): Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.939818 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.930531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:45621 (local address 127.0.0.1:45084)
0619 02:45:05.930773 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.930824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:05.930974 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:05.931682 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:05.931690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:05.931728 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:05.932028 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:05.932043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.934173 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.934182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:05.936665 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:05.936690 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.938895 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.938903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:05.939189 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:05.939231 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:45:05.939358 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 02:45:05.939505 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":22}
I20260619 02:45:05.940447 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.927564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.7.246.125:45621)
0619 02:45:05.930674 (+ 3110us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:05.930683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:05.931101 (+ 418us) server_negotiation.cc:407] Connection header received
0619 02:45:05.931180 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:05.931184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:05.931261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:05.931357 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:05.932250 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.933932 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.936936 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.938554 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.938684 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:05.938796 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:05.939903 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0619 02:45:05.940052 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2938,"thread_start_us":2679,"threads_started":1}
W20260619 02:45:05.939816 15071 leader_election.cc:334] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 246a54d557974c13a38a4faee724048b (127.7.246.126:36239): Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.941458 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22aeb734eae949f1aaee5d2ae407820a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" is_pre_election: true
I20260619 02:45:05.941861 14908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22aeb734eae949f1aaee5d2ae407820a in term 1.
I20260619 02:45:05.942438 15071 leader_election.cc:302] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22aeb734eae949f1aaee5d2ae407820a, 550ed9b2244d4d0ebcc10a7c63e7df82; no voters: 246a54d557974c13a38a4faee724048b
I20260619 02:45:05.942831 15627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:45:05.943006 15627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 246a54d557974c13a38a4faee724048b)
I20260619 02:45:05.943152 15627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:45:05.944988 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.916909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.7.246.124:44117)
0619 02:45:05.937477 (+ 20568us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:05.937484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:05.937500 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:05.937579 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:05.937584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:05.937655 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:05.937757 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:05.938732 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.940053 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.942494 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.943903 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.944021 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:05.944112 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:05.944507 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 02:45:05.944618 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20297,"thread_start_us":20150,"threads_started":1}
I20260619 02:45:05.945019 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.916708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:44117 (local address 127.0.0.1:45336)
0619 02:45:05.930358 (+ 13650us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.930384 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:05.930556 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:05.938007 (+ 7451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:05.938013 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:05.938070 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:05.938522 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:05.938540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.940180 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.940187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:05.942330 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:05.942345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:05.944103 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:05.944110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:05.944319 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:05.944360 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:45:05.944477 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:45:05.944628 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13494,"thread_start_us":13148,"threads_started":1}
I20260619 02:45:05.945783 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "22aeb734eae949f1aaee5d2ae407820a" is_pre_election: true
I20260619 02:45:05.946493 15627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } }
I20260619 02:45:05.947006 15097 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 550ed9b2244d4d0ebcc10a7c63e7df82 in current term 2: Already voted for candidate 22aeb734eae949f1aaee5d2ae407820a in this term.
I20260619 02:45:05.947222 15627 leader_election.cc:288] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 election: Requested vote from peers 246a54d557974c13a38a4faee724048b (127.7.246.126:36239), 550ed9b2244d4d0ebcc10a7c63e7df82 (127.7.246.125:45621)
I20260619 02:45:05.947738 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.947331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44518)
0619 02:45:05.947483 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.947577 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:45:05.947741 14885 leader_election.cc:302] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 550ed9b2244d4d0ebcc10a7c63e7df82; no voters: 22aeb734eae949f1aaee5d2ae407820a, 246a54d557974c13a38a4faee724048b
I20260619 02:45:05.947944 14908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22aeb734eae949f1aaee5d2ae407820a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82"
I20260619 02:45:05.948156 14908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:45:05.948767 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.948369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44534)
0619 02:45:05.948533 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.948640 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:45:05.948983 15071 leader_election.cc:334] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 246a54d557974c13a38a4faee724048b (127.7.246.126:36239): Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
I20260619 02:45:05.950564 14908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22aeb734eae949f1aaee5d2ae407820a in term 2.
I20260619 02:45:05.950748 15626 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:45:05.951071 15071 leader_election.cc:302] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22aeb734eae949f1aaee5d2ae407820a, 550ed9b2244d4d0ebcc10a7c63e7df82; no voters: 246a54d557974c13a38a4faee724048b
I20260619 02:45:05.951365 15627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:45:05.951829 15627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [term 2 LEADER]: Becoming Leader. State: Replica: 22aeb734eae949f1aaee5d2ae407820a, State: Running, Role: LEADER
I20260619 02:45:05.952209 15627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } }
I20260619 02:45:05.954355 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22aeb734eae949f1aaee5d2ae407820a. Latest consensus state: current_term: 2 leader_uuid: "22aeb734eae949f1aaee5d2ae407820a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:05.954679 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:05.955518 15638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:05.959417 15638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:05.960234 15638 catalog_manager.cc:1259] Loaded cluster ID: 61838aebcd5246fe86c6a4b05f5c3e0e
I20260619 02:45:05.960300 15638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:05.961369 15638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:05.962201 15638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Loaded TSK: 0
I20260619 02:45:05.963045 15638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:45:05.970475 15086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:05.977672 15086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:05.982110 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.981678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44548)
0619 02:45:05.981852 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.981952 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:45:05.982679 14908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [term 2 FOLLOWER]: Refusing update from remote peer 22aeb734eae949f1aaee5d2ae407820a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 02:45:05.983280 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.982848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44562)
0619 02:45:05.983004 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.983112 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:45:05.983631 15636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [LEADER]: Connected to new peer: Peer: permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 02:45:05.984100 15071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a -> Peer 246a54d557974c13a38a4faee724048b (127.7.246.126:36239): Couldn't send request to peer 246a54d557974c13a38a4faee724048b. Status: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:45:05.986899 15626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22aeb734eae949f1aaee5d2ae407820a. Latest consensus state: current_term: 2 leader_uuid: "22aeb734eae949f1aaee5d2ae407820a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:05.987185 15626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:05.988806 15636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "22aeb734eae949f1aaee5d2ae407820a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:05.989161 15636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:05.990061 15626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "22aeb734eae949f1aaee5d2ae407820a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "246a54d557974c13a38a4faee724048b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36239 } } peers { permanent_uuid: "550ed9b2244d4d0ebcc10a7c63e7df82" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 45621 } attrs { promote: false } } peers { permanent_uuid: "22aeb734eae949f1aaee5d2ae407820a" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 44117 } attrs { promote: false } } }
I20260619 02:45:05.990350 15626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ed9b2244d4d0ebcc10a7c63e7df82 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:45:06.006906 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.998356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.7.246.65:39293)
0619 02:45:06.001652 (+ 3296us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.001657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.001674 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:45:06.001731 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.001735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.001793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.001900 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:06.002281 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.003007 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.005102 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.005952 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.006099 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.006213 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.006466 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.006603 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":519,"threads_started":1}
I20260619 02:45:06.006898 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.998256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:39293 (local address 127.0.0.1:35246)
0619 02:45:05.998668 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.998689 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:05.998794 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:06.001931 (+ 3137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:06.001936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:06.001960 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:06.002147 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.002156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.003143 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.003149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.004929 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.004940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.006109 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.006115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.006312 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.006337 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:45:06.006443 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:45:06.006581 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260619 02:45:06.007982 15202 tablet_service.cc:1467] Processing CreateTablet for tablet e0a1a9154a67492188f386a6f4621bee (DEFAULT_TABLE table=test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:06.008502 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a1a9154a67492188f386a6f4621bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:06.014549 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.999316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.7.246.67:46829)
0619 02:45:05.999743 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:05.999748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.007805 (+ 8057us) server_negotiation.cc:407] Connection header received
0619 02:45:06.007893 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.007898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.008004 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.008527 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:06.008704 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.009749 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.012056 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.013290 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.013445 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.013587 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.014073 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.014218 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":249,"threads_started":1}
I20260619 02:45:06.014602 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.999238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:46829 (local address 127.0.0.1:38172)
0619 02:45:06.007285 (+ 8047us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.007303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:06.007684 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:06.008151 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:06.008155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:06.008203 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:06.008380 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.008391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.009898 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.009904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.011829 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.011844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.013490 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.013502 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.013800 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.013904 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 02:45:06.014016 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:45:06.014171 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7949,"mutex_wait_us":51,"thread_start_us":12819,"threads_started":1}
I20260619 02:45:06.015215 15649 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Bootstrap starting.
I20260619 02:45:06.017411 15649 tablet_bootstrap.cc:654] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:06.017959 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.998668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39321 (local address 127.0.0.1:34162)
0619 02:45:05.998982 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:05.999004 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:05.999073 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:06.011675 (+ 12602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:06.011680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:06.011710 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:06.012130 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.012142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.013259 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.013277 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.015469 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.015491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.017191 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.017196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.017319 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.017343 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:45:06.017454 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:45:06.017639 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260619 02:45:06.018268 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:05.998992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.7.246.66:39321)
0619 02:45:06.011368 (+ 12376us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.011372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.011390 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:45:06.011451 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.011455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.011515 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.011654 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:06.012342 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.013108 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.015691 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.016951 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.017150 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.017323 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.017756 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.017908 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12249,"thread_start_us":12137,"threads_started":1}
I20260619 02:45:06.018196 15489 tablet_service.cc:1467] Processing CreateTablet for tablet e0a1a9154a67492188f386a6f4621bee (DEFAULT_TABLE table=test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:06.018718 15489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a1a9154a67492188f386a6f4621bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:06.019130 15348 tablet_service.cc:1467] Processing CreateTablet for tablet e0a1a9154a67492188f386a6f4621bee (DEFAULT_TABLE table=test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:06.019476 15348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a1a9154a67492188f386a6f4621bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:06.023545 15649 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: No bootstrap required, opened a new log
I20260619 02:45:06.023727 15649 ts_tablet_manager.cc:1397] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260619 02:45:06.024852 15651 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Bootstrap starting.
I20260619 02:45:06.024648 15649 raft_consensus.cc:348] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.024953 15649 raft_consensus.cc:374] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:06.025106 15649 raft_consensus.cc:729] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7478f303cc94ce6af89c965b493a3a0, State: Initialized, Role: FOLLOWER
I20260619 02:45:06.025421 15649 consensus_queue.cc:260] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.026736 15651 tablet_bootstrap.cc:654] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:06.026875 15653 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Bootstrap starting.
I20260619 02:45:06.027822 15649 ts_tablet_manager.cc:1428] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 02:45:06.029593 15653 tablet_bootstrap.cc:654] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:06.033638 15267 heartbeater.cc:503] Master 127.7.246.124:44117 was elected leader, sending a full tablet report...
I20260619 02:45:06.038512 15651 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: No bootstrap required, opened a new log
I20260619 02:45:06.038569 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.028432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:39209)
0619 02:45:06.038123 (+ 9691us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.038297 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9500,"thread_start_us":9359,"threads_started":1}
I20260619 02:45:06.038878 15651 ts_tablet_manager.cc:1397] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260619 02:45:06.039849 15651 raft_consensus.cc:348] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.040174 15651 raft_consensus.cc:374] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:06.040309 15651 raft_consensus.cc:729] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3f228c8d1c425184bc717e23396549, State: Initialized, Role: FOLLOWER
I20260619 02:45:06.040338 15653 tablet_bootstrap.cc:492] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: No bootstrap required, opened a new log
I20260619 02:45:06.040477 15653 ts_tablet_manager.cc:1397] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260619 02:45:06.040623 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.040241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:33405)
0619 02:45:06.040381 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.040478 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:45:06.040689 15651 consensus_queue.cc:260] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.041364 15653 raft_consensus.cc:348] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.041636 15651 ts_tablet_manager.cc:1428] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:06.041694 15653 raft_consensus.cc:374] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:06.041785 15653 raft_consensus.cc:729] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad410b9ad6704ad095daa5fbbcf3088c, State: Initialized, Role: FOLLOWER
I20260619 02:45:06.042048 15653 consensus_queue.cc:260] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.043617 15653 ts_tablet_manager.cc:1428] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:06.043707 15413 heartbeater.cc:503] Master 127.7.246.124:44117 was elected leader, sending a full tablet report...
I20260619 02:45:06.045439 15554 heartbeater.cc:503] Master 127.7.246.124:44117 was elected leader, sending a full tablet report...
I20260619 02:45:06.050619 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.045213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:43663)
0619 02:45:06.050147 (+ 4934us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.050346 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4794,"thread_start_us":4665,"threads_started":1}
I20260619 02:45:06.051162 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.042177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:50545)
0619 02:45:06.050058 (+ 7881us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.050244 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7733,"thread_start_us":7598,"threads_started":1}
I20260619 02:45:06.052191 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.051806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:50473)
0619 02:45:06.051969 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.052054 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 02:45:06.052803 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.051832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:60689)
0619 02:45:06.052591 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.052670 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260619 02:45:06.078969 15657 raft_consensus.cc:482] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:06.079308 15657 raft_consensus.cc:504] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.081007 15657 leader_election.cc:288] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293), ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:06.087307 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.081377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:49249 (local address 127.7.246.65:39293)
0619 02:45:06.081603 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.081611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.081633 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:45:06.081819 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.081825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.081905 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.081999 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:06.082555 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.083448 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.085362 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.086322 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.086440 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.086500 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.086872 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.087023 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:45:06.087311 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.081226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:39293 (local address 127.7.246.66:49249)
0619 02:45:06.081442 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.081467 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:06.081653 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:06.082038 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:06.082041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:06.082063 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:06.082378 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.082390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.083601 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.083614 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.085203 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.085215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.086506 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.086512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.086702 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.086739 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:45:06.086848 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:45:06.087013 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 02:45:06.087872 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.081769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:58689 (local address 127.7.246.67:46829)
0619 02:45:06.081943 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.081948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.082139 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:45:06.082285 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.082293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.082389 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.082494 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:06.083114 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.084092 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.085853 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.086844 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.086993 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.087058 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.087496 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.087636 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:45:06.088016 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a1a9154a67492188f386a6f4621bee" candidate_uuid: "ba3f228c8d1c425184bc717e23396549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7478f303cc94ce6af89c965b493a3a0" is_pre_election: true
I20260619 02:45:06.088276 15222 raft_consensus.cc:2454] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba3f228c8d1c425184bc717e23396549 in term 0.
I20260619 02:45:06.088549 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.081283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:46829 (local address 127.7.246.66:58689)
0619 02:45:06.082026 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.082044 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:06.082181 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:06.082552 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:06.082556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:06.082579 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:06.082914 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.082940 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.084227 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.084231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.085700 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:06.085714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.087064 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.087070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:06.087292 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.087336 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:45:06.088085 (+ 749us) client_negotiation.cc:240] Negotiation successful
0619 02:45:06.088235 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":524,"threads_started":1}
I20260619 02:45:06.088958 15304 leader_election.cc:302] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba3f228c8d1c425184bc717e23396549, e7478f303cc94ce6af89c965b493a3a0; no voters:
I20260619 02:45:06.089430 15657 raft_consensus.cc:2781] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:06.089399 15509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a1a9154a67492188f386a6f4621bee" candidate_uuid: "ba3f228c8d1c425184bc717e23396549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" is_pre_election: true
I20260619 02:45:06.089594 15657 raft_consensus.cc:482] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:06.089707 15657 raft_consensus.cc:3037] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:06.089877 15509 raft_consensus.cc:2454] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba3f228c8d1c425184bc717e23396549 in term 0.
I20260619 02:45:06.092130 15657 raft_consensus.cc:504] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.092772 15657 leader_election.cc:288] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [CANDIDATE]: Term 1 election: Requested vote from peers e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293), ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829)
I20260619 02:45:06.093312 15509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a1a9154a67492188f386a6f4621bee" candidate_uuid: "ba3f228c8d1c425184bc717e23396549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad410b9ad6704ad095daa5fbbcf3088c"
I20260619 02:45:06.093312 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a1a9154a67492188f386a6f4621bee" candidate_uuid: "ba3f228c8d1c425184bc717e23396549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7478f303cc94ce6af89c965b493a3a0"
I20260619 02:45:06.093518 15509 raft_consensus.cc:3037] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:06.093530 15222 raft_consensus.cc:3037] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:06.096519 15222 raft_consensus.cc:2454] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3f228c8d1c425184bc717e23396549 in term 1.
I20260619 02:45:06.096519 15509 raft_consensus.cc:2454] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3f228c8d1c425184bc717e23396549 in term 1.
I20260619 02:45:06.096971 15304 leader_election.cc:302] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ad410b9ad6704ad095daa5fbbcf3088c, ba3f228c8d1c425184bc717e23396549, e7478f303cc94ce6af89c965b493a3a0; no voters:
I20260619 02:45:06.097334 15657 raft_consensus.cc:2781] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:06.097810 15657 raft_consensus.cc:686] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 1 LEADER]: Becoming Leader. State: Replica: ba3f228c8d1c425184bc717e23396549, State: Running, Role: LEADER
I20260619 02:45:06.098413 15657 consensus_queue.cc:237] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } }
I20260619 02:45:06.098893 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.098424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:38697)
0619 02:45:06.098600 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.098738 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:45:06.101822 15086 catalog_manager.cc:5495] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba3f228c8d1c425184bc717e23396549 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "ba3f228c8d1c425184bc717e23396549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba3f228c8d1c425184bc717e23396549" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 } health_report { overall_health: UNKNOWN } } }
I20260619 02:45:06.119086 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.113911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.7.246.66:39321)
0619 02:45:06.114120 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.114125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.114143 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:45:06.114323 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.114344 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.114418 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.114561 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:06.114894 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.115568 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.116850 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.117269 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.117412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.117899 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:06.118512 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:06.118668 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.118745 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.118823 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:45:06.123193 15222 raft_consensus.cc:1261] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Refusing update from remote peer ba3f228c8d1c425184bc717e23396549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:06.123539 15509 raft_consensus.cc:1261] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Refusing update from remote peer ba3f228c8d1c425184bc717e23396549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:06.123941 15662 consensus_queue.cc:1035] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7478f303cc94ce6af89c965b493a3a0" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:06.124351 15657 consensus_queue.cc:1035] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad410b9ad6704ad095daa5fbbcf3088c" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:06.127106 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.126533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:43031)
0619 02:45:06.126751 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.126863 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 02:45:06.127656 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.126531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:52351)
0619 02:45:06.127375 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.127486 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
I20260619 02:45:06.132546 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.131964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:51061)
0619 02:45:06.132273 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.132356 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260619 02:45:06.133728 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.131774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.65:39893)
0619 02:45:06.133394 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.133510 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1456}
I20260619 02:45:06.134640 15665 mvcc.cc:205] Tried to move back new op lower bound from 7298404786673094656 to 7298404786583134208. Current Snapshot: MvccSnapshot[applied={T|T < 7298404786673094656 or (T in {7298404786673094656})}]
I20260619 02:45:06.136583 15666 mvcc.cc:205] Tried to move back new op lower bound from 7298404786673094656 to 7298404786583134208. Current Snapshot: MvccSnapshot[applied={T|T < 7298404786673094656}]
I20260619 02:45:06.140627 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.140121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.67:37849)
0619 02:45:06.140372 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.140458 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 02:45:06.141206 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.140512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.7.246.66:35559)
0619 02:45:06.140943 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.141024 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
I20260619 02:45:06.151206 15087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:06.151551 15087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:06.157404 15087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22aeb734eae949f1aaee5d2ae407820a: Sending DeleteTablet for 3 replicas of tablet e0a1a9154a67492188f386a6f4621bee
I20260619 02:45:06.158556 15348 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a1a9154a67492188f386a6f4621bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:06 UTC) from {username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:34162
I20260619 02:45:06.158550 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a1a9154a67492188f386a6f4621bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:06 UTC) from {username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:35246
I20260619 02:45:06.159022 15489 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a1a9154a67492188f386a6f4621bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:06 UTC) from {username='oryx', principal='oryx/127.7.246.124@KRBTEST.COM'} at 127.0.0.1:38172
I20260619 02:45:06.159754 15677 tablet_replica.cc:330] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: stopping tablet replica
I20260619 02:45:06.159993 15677 raft_consensus.cc:2229] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:06.160419 15677 raft_consensus.cc:2258] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:06.163355 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15122
I20260619 02:45:06.169616 15677 ts_tablet_manager.cc:1905] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:06.171483 15676 tablet_replica.cc:330] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: stopping tablet replica
I20260619 02:45:06.171667 15676 raft_consensus.cc:2229] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:06.171929 15676 raft_consensus.cc:2258] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:06.173080 15676 ts_tablet_manager.cc:1905] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:06.181654 15676 ts_tablet_manager.cc:1918] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:06.181833 15676 log.cc:1196] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/e0a1a9154a67492188f386a6f4621bee
I20260619 02:45:06.182394 15676 ts_tablet_manager.cc:1939] T e0a1a9154a67492188f386a6f4621bee P e7478f303cc94ce6af89c965b493a3a0: Deleting consensus metadata
I20260619 02:45:06.183738 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.164896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.7.246.65:39293)
0619 02:45:06.165106 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.165112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.165131 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:45:06.165189 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.165196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.165273 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.165387 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:06.167319 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.168602 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.170675 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.171346 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.171445 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.175484 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:06.175503 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:06.175527 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:06.175584 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:06.179749 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.180687 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.180700 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.180706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.180863 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.181347 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.181352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.181355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.182081 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:06.182374 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.183062 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.183347 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:45:06.183786 15074 catalog_manager.cc:4841] TS e7478f303cc94ce6af89c965b493a3a0 (127.7.246.65:39293): tablet e0a1a9154a67492188f386a6f4621bee (table test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]) successfully deleted
I20260619 02:45:06.184758 15677 ts_tablet_manager.cc:1918] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:06.185141 15262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:06.186180 15677 log.cc:1196] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/e0a1a9154a67492188f386a6f4621bee
I20260619 02:45:06.186672 15677 ts_tablet_manager.cc:1939] T e0a1a9154a67492188f386a6f4621bee P ad410b9ad6704ad095daa5fbbcf3088c: Deleting consensus metadata
I20260619 02:45:06.187991 15073 catalog_manager.cc:4841] TS ad410b9ad6704ad095daa5fbbcf3088c (127.7.246.67:46829): tablet e0a1a9154a67492188f386a6f4621bee (table test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]) successfully deleted
I20260619 02:45:06.189193 15675 tablet_replica.cc:330] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: stopping tablet replica
I20260619 02:45:06.189486 15675 raft_consensus.cc:2229] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:06.190148 15675 raft_consensus.cc:2258] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:06.191361 15675 ts_tablet_manager.cc:1905] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:06.198770 15675 ts_tablet_manager.cc:1918] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:06.198946 15675 log.cc:1196] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/e0a1a9154a67492188f386a6f4621bee
I20260619 02:45:06.199483 15675 ts_tablet_manager.cc:1939] T e0a1a9154a67492188f386a6f4621bee P ba3f228c8d1c425184bc717e23396549: Deleting consensus metadata
I20260619 02:45:06.200790 15071 catalog_manager.cc:4841] TS ba3f228c8d1c425184bc717e23396549 (127.7.246.66:39321): tablet e0a1a9154a67492188f386a6f4621bee (table test-table [id=19a5ac11c4114bb4b3b5e743b34e723f]) successfully deleted
I20260619 02:45:06.284658 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15122
I20260619 02:45:06.302273 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15279
I20260619 02:45:06.318050 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.303088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.7.246.66:39321)
0619 02:45:06.303283 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.303288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.303431 (+ 143us) server_negotiation.cc:407] Connection header received
0619 02:45:06.303598 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.303602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.303686 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.303778 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:06.305478 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.306284 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.308112 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.308567 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.308669 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.311945 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:06.311960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:06.311987 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:06.312021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:06.315106 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.315939 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.315946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.315948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.316107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.316488 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.316491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.316493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.316811 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:06.317053 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.317368 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.317523 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:45:06.318866 15408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:06.435060 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15279
I20260619 02:45:06.450685 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15426
I20260619 02:45:06.464223 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.451492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.7.246.67:46829)
0619 02:45:06.451668 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:06.451672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:06.451688 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:06.451867 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:06.451871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:06.451931 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:06.452042 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:06.453479 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.454267 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.455579 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:06.455936 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:06.456024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:06.458766 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:06.458780 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:06.458797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:06.458828 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:06.461672 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.462369 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.462375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.462377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.462510 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:06.462904 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:06.462907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:06.462909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:06.463238 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:06.463438 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:06.463691 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 02:45:06.463873 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:45:06.464910 15549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:06.484700 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.484259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44574)
0619 02:45:06.484436 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.484538 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 02:45:06.486129 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:06.485744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36239 (local address 127.0.0.1:44590)
0619 02:45:06.485920 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:06.486011 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:36239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 02:45:06.585176 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15426
I20260619 02:45:06.608258 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14760
I20260619 02:45:06.609390 14928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:06.723327 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14760
I20260619 02:45:06.743680 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 14861
I20260619 02:45:06.744791 15117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:06.864313 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 14861
2026-06-19T02:45:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9484 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:45:06 dist-test-slave-9tkz krb5kdc[15687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:45:06 dist-test-slave-9tkz krb5kdc[15687](info): set up 2 sockets
Jun 19 02:45:06 dist-test-slave-9tkz krb5kdc[15687](info): commencing operation
krb5kdc: starting...
W20260619 02:45:07.970517 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:45:08 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab.
Entry for principal oryx/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab.
I20260619 02:45:08.188529 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:39729
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab
--principal=oryx/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:08.502754 15700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:08.503036 15700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:08.503229 15700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:08.510510 15700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:08.510622 15700 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:45:08.510704 15700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:08.510813 15700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:08.510893 15700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:08.510982 15700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:08.520579 15700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:39729
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.126.keytab
--principal=oryx/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:08.521911 15700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:08.523280 15700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:08.535215 15708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:08.535250 15706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:08.535180 15705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:08.536499 15700 server_base.cc:1034] running on GCE node
I20260619 02:45:08.537274 15700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:08.538801 15700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:08.540037 15700 hybrid_clock.cc:648] HybridClock initialized: now 1781837108539966 us; error 54 us; skew 500 ppm
I20260619 02:45:08.540385 15700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:08 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837108, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:08.543540 15700 init.cc:377] Logged in from keytab as oryx/127.7.246.126@KRBTEST.COM (short username oryx)
I20260619 02:45:08.545092 15700 webserver.cc:460] Webserver started at http://127.7.246.126:42611/ using document root <none> and password file <none>
I20260619 02:45:08.545858 15700 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:08.545986 15700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:08.546303 15700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:08.549203 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "8a7a109df68a42e49c40f7c1a9595a98"
format_stamp: "Formatted at 2026-06-19 02:45:08 on dist-test-slave-9tkz"
server_key: "761c4520f3fdc58fa5f73890328d4c45"
server_key_iv: "595bed87e2f7c02a1439a4a41564b391"
server_key_version: "encryptionkey@0"
I20260619 02:45:08.550034 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "8a7a109df68a42e49c40f7c1a9595a98"
format_stamp: "Formatted at 2026-06-19 02:45:08 on dist-test-slave-9tkz"
server_key: "761c4520f3fdc58fa5f73890328d4c45"
server_key_iv: "595bed87e2f7c02a1439a4a41564b391"
server_key_version: "encryptionkey@0"
I20260619 02:45:08.555073 15700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:45:08.558651 15715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:08.560093 15700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:45:08.560276 15700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "8a7a109df68a42e49c40f7c1a9595a98"
format_stamp: "Formatted at 2026-06-19 02:45:08 on dist-test-slave-9tkz"
server_key: "761c4520f3fdc58fa5f73890328d4c45"
server_key_iv: "595bed87e2f7c02a1439a4a41564b391"
server_key_version: "encryptionkey@0"
I20260619 02:45:08.560500 15700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:08.592013 15700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:08.596664 15700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:08.596947 15700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:08.618116 15700 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:39729
I20260619 02:45:08.618124 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:39729 every 8 connection(s)
I20260619 02:45:08.619501 15700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:08.623170 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:08.629403 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15700
I20260619 02:45:08.629680 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:45:08.630505 8153 external_mini_cluster.cc:1428] Setting key 5c366f0ad9d7efa58fdd12ba18a7666f
I20260619 02:45:08.634326 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Bootstrap starting.
I20260619 02:45:08.637823 15777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:08.638943 15777 log.cc:824] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:08.642195 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: No bootstrap required, opened a new log
Jun 19 02:45:08 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:08.651562 15777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } }
I20260619 02:45:08.652052 15777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:08.652179 15777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7a109df68a42e49c40f7c1a9595a98, State: Initialized, Role: FOLLOWER
I20260619 02:45:08.652905 15777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } }
I20260619 02:45:08.653148 15777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:08.653316 15777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:08.653512 15777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:08.654151 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:08.632701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.7.246.126:39729)
0619 02:45:08.633594 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:08.633611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:08.633663 (+ 52us) server_negotiation.cc:407] Connection header received
0619 02:45:08.635442 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:08.635465 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:08.635877 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:08.636208 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:08.638032 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:08.639623 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:08.641062 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:08.642104 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:08.642312 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:08.646494 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:08.646502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:08.646518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:08.646552 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:08.649704 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:08.650556 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:08.650562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:08.650569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:08.650723 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:08.651128 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:08.651132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:08.651134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:08.651659 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:08.651850 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:08.652436 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 02:45:08.652829 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":423,"threads_started":1}
I20260619 02:45:08.655939 15777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } }
I20260619 02:45:08.656442 15777 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7a109df68a42e49c40f7c1a9595a98; no voters:
I20260619 02:45:08.657081 15777 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:45:08.657183 15782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:08.657979 15782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7a109df68a42e49c40f7c1a9595a98, State: Running, Role: LEADER
I20260619 02:45:08.658829 15782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } }
I20260619 02:45:08.659291 15777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:08.661834 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7a109df68a42e49c40f7c1a9595a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } } }
I20260619 02:45:08.662344 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:08.662660 15784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7a109df68a42e49c40f7c1a9595a98. Latest consensus state: current_term: 1 leader_uuid: "8a7a109df68a42e49c40f7c1a9595a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7a109df68a42e49c40f7c1a9595a98" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39729 } } }
I20260619 02:45:08.662995 15784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:08.664333 15791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:08.668958 15791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:08.677173 15791 catalog_manager.cc:1347] Generated new cluster ID: bca77e4f6f1f4f87990feeef3ab92604
I20260619 02:45:08.677294 15791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:08.709053 15791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:45:08.710934 15791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:08.723017 15791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Generated new TSK 0
I20260619 02:45:08.723866 15791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
I20260619 02:45:08.777344 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:39729
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:09.102063 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:09.102353 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:09.102524 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:09.109799 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:09.109910 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:09.110070 15803 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:45:09.110185 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:09.120385 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:39729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:09.121747 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:09.123049 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:09.135731 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:09.135735 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:09.135900 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:09.136829 15803 server_base.cc:1034] running on GCE node
I20260619 02:45:09.137538 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:09.138572 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:09.139837 15803 hybrid_clock.cc:648] HybridClock initialized: now 1781837109139765 us; error 41 us; skew 500 ppm
I20260619 02:45:09.140157 15803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:09 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837109, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:09.148020 15803 init.cc:377] Logged in from keytab as oryx/127.7.246.65@KRBTEST.COM (short username oryx)
I20260619 02:45:09.149447 15803 webserver.cc:460] Webserver started at http://127.7.246.65:44669/ using document root <none> and password file <none>
I20260619 02:45:09.150157 15803 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:09.150290 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:09.150641 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:09.153643 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "8338ba0738f94e768c4d40a8bc6dc986"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "43bbfa28abab8b237ce1e70fdcdd68e8"
server_key_iv: "0a74797d414d84ffb0760f3f48e439de"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.154462 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "8338ba0738f94e768c4d40a8bc6dc986"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "43bbfa28abab8b237ce1e70fdcdd68e8"
server_key_iv: "0a74797d414d84ffb0760f3f48e439de"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.159853 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:45:09.163338 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:09.165028 15803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:09.165226 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "8338ba0738f94e768c4d40a8bc6dc986"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "43bbfa28abab8b237ce1e70fdcdd68e8"
server_key_iv: "0a74797d414d84ffb0760f3f48e439de"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.165510 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:09.185966 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:09.190088 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:09.190375 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:09.192940 15803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:09.193050 15803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:09.193226 15803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:09.193322 15803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:45:09 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837109, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:09.218961 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.193733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:39937 (local address 127.7.246.126:39729)
0619 02:45:09.194222 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:09.194229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:09.195487 (+ 1258us) server_negotiation.cc:407] Connection header received
0619 02:45:09.197576 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:09.197582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:09.197676 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:09.197801 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:09.200686 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.201475 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.204381 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.205113 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.205243 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.210366 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:09.210373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:09.210377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:09.210440 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:09.213634 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.214748 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.214758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.214773 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.214960 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.215862 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.215870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.215873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.216335 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:09.216486 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:09.218049 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0619 02:45:09.218457 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":268,"threads_started":1}
I20260619 02:45:09.221897 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.194450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.65:39937)
0619 02:45:09.195289 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:09.195337 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:09.197240 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:09.198302 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:09.198318 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:09.199329 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:09.200447 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.200486 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.201693 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.201697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.204203 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.204218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.205634 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.205642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.205995 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.206912 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:09.206941 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:09.210049 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:09.213812 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.213824 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.213844 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.214590 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.215148 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.215155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.215162 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.215461 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.216615 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:09.216627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:09.217468 (+ 841us) client_negotiation.cc:769] Sending connection context
0619 02:45:09.219314 (+ 1846us) client_negotiation.cc:240] Negotiation successful
0619 02:45:09.220019 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260619 02:45:09.267894 15803 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33641
I20260619 02:45:09.267921 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33641 every 8 connection(s)
I20260619 02:45:09.269196 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.237815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:38463 (local address 127.7.246.126:39729)
0619 02:45:09.238055 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:09.238065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:09.253330 (+ 15265us) server_negotiation.cc:407] Connection header received
0619 02:45:09.253521 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:09.253529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:09.253660 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:09.253787 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:09.255273 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.256571 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.258234 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.258981 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.259116 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.262379 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:09.262388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:09.262395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:09.262444 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:09.266070 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.266897 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.266905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.266909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.267073 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.267517 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.267524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.267527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.267892 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:09.268036 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:09.268587 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 02:45:09.268799 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:45:09.269354 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.237679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.65:38463)
0619 02:45:09.253162 (+ 15483us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:09.253187 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:09.253379 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:09.254018 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:09.254025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:09.254804 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:09.255105 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.255114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.256756 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.256765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.258087 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.258099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.259155 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.259162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.259335 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.260564 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:09.260595 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:09.262116 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:09.266182 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.266188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.266191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.266726 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.267174 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.267179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.267183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.267353 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.268096 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:09.268103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:09.268317 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:45:09.268483 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 02:45:09.268851 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15242,"thread_start_us":15767,"threads_started":1}
I20260619 02:45:09.269832 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:09.270959 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15803
I20260619 02:45:09.271178 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:45:09.271963 8153 external_mini_cluster.cc:1428] Setting key 6991d0028181a10956cbcd25f6f742c2
I20260619 02:45:09.272040 15939 heartbeater.cc:347] Connected to a master server at 127.7.246.126:39729
I20260619 02:45:09.272262 15939 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:09.273355 15939 heartbeater.cc:511] Master 127.7.246.126:39729 requested a full tablet report, sending...
I20260619 02:45:09.275542 15732 ts_manager.cc:194] Registered new tserver with Master: 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641)
I20260619 02:45:09.276938 15732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:39937
WARNING: no policy specified for oryx/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
I20260619 02:45:09.317153 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:39729
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:09.634861 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:09.635191 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:09.635375 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:09.642203 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:09.642346 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:09.642535 15943 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:45:09.642694 15943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:09.652567 15943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:39729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:09.653892 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:09.655143 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:09.667972 15948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:09.668002 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:09.668025 15949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:09.669013 15943 server_base.cc:1034] running on GCE node
I20260619 02:45:09.669749 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:09.670730 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:09.671954 15943 hybrid_clock.cc:648] HybridClock initialized: now 1781837109671871 us; error 55 us; skew 500 ppm
I20260619 02:45:09.672250 15943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:09 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837109, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:09.675362 15943 init.cc:377] Logged in from keytab as oryx/127.7.246.66@KRBTEST.COM (short username oryx)
I20260619 02:45:09.676774 15943 webserver.cc:460] Webserver started at http://127.7.246.66:38515/ using document root <none> and password file <none>
I20260619 02:45:09.677435 15943 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:09.677547 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:09.677901 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:09.680763 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "1a60c00596015611dd2a6cba9f6d1f17"
server_key_iv: "7dad48f436f0864eeedad5f266ca9542"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.681586 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "1a60c00596015611dd2a6cba9f6d1f17"
server_key_iv: "7dad48f436f0864eeedad5f266ca9542"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.686733 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 02:45:09.689987 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:09.691344 15943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:45:09.691521 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
format_stamp: "Formatted at 2026-06-19 02:45:09 on dist-test-slave-9tkz"
server_key: "1a60c00596015611dd2a6cba9f6d1f17"
server_key_iv: "7dad48f436f0864eeedad5f266ca9542"
server_key_version: "encryptionkey@0"
I20260619 02:45:09.691746 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:09.711055 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:09.715134 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:09.715427 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:09.718178 15943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:09.718286 15943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:09.718466 15943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:09.718580 15943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 02:45:09 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837109, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:09.739042 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.718667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:45235 (local address 127.7.246.126:39729)
0619 02:45:09.718858 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:09.718865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:09.720246 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 02:45:09.722214 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:09.722220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:09.722334 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:09.722440 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:09.725034 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.725946 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.728523 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.729111 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.729241 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.733187 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:09.733192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:09.733197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:09.733236 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:09.736022 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.736670 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.736677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.736681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.736798 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.737190 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.737196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.737199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.737470 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:09.737635 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:09.738504 (+ 869us) server_negotiation.cc:299] Negotiation successful
0619 02:45:09.738710 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:45:09.740036 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.719319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.66:45235)
0619 02:45:09.720098 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:09.720138 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:09.721957 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:09.722694 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:09.722711 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:09.723939 (+ 1228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:09.724847 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.724876 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.726089 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.726094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.728286 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.728321 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.729349 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.729358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.729736 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.730590 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:09.730618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:09.732984 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:09.736151 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.736160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.736175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.736558 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.736898 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.736903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.736907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.737081 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.737604 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:09.737611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:09.738053 (+ 442us) client_negotiation.cc:769] Sending connection context
0619 02:45:09.738429 (+ 376us) client_negotiation.cc:240] Negotiation successful
0619 02:45:09.738922 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":349,"threads_started":1}
I20260619 02:45:09.762243 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.746373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:45511 (local address 127.7.246.126:39729)
0619 02:45:09.746580 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:09.746589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:09.747302 (+ 713us) server_negotiation.cc:407] Connection header received
0619 02:45:09.747493 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:09.747497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:09.747555 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:09.747633 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:09.749226 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.750127 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.752179 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.752579 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.752735 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.755396 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:09.755402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:09.755407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:09.755450 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:09.759140 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.760233 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.760237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.760239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.760357 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:09.760811 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:09.760814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:09.760817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:09.761134 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:09.761240 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:09.761848 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 02:45:09.762032 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:45:09.762850 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:09.746349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.66:45511)
0619 02:45:09.747153 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:09.747178 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:09.747370 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:09.747696 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:09.747704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:09.748436 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:09.748959 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.749037 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.750294 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.750300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.752024 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:09.752048 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:09.753011 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:09.753016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:09.753226 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:09.754029 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:09.754076 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:09.755208 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:09.759325 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.759359 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.759366 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.760102 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.760472 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:09.760480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:09.760486 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:09.760684 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:09.761249 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:09.761257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:09.761509 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 02:45:09.761742 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 02:45:09.762195 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":384,"threads_started":1}
I20260619 02:45:09.774715 15943 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:44971
I20260619 02:45:09.774739 16077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:44971 every 8 connection(s)
I20260619 02:45:09.776120 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:09.777709 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 15943
I20260619 02:45:09.777896 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:09.778342 16078 heartbeater.cc:347] Connected to a master server at 127.7.246.126:39729
I20260619 02:45:09.778436 8153 external_mini_cluster.cc:1428] Setting key 304aea2fbc2b7c3bf7004690b547353d
I20260619 02:45:09.778590 16078 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:09.779331 16078 heartbeater.cc:511] Master 127.7.246.126:39729 requested a full tablet report, sending...
I20260619 02:45:09.780637 15732 ts_manager.cc:194] Registered new tserver with Master: 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971)
I20260619 02:45:09.781440 15732 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:45235
WARNING: no policy specified for oryx/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
I20260619 02:45:09.824663 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:39729
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:10.150184 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:10.150507 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:10.150673 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:10.157714 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:10.157850 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:10.158030 16082 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:45:10.158171 16082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:10.168143 16082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.7.246.126:39729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:10.169523 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:10.170971 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:10.183706 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:10.183733 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:10.184273 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:10.185134 16082 server_base.cc:1034] running on GCE node
I20260619 02:45:10.185818 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:10.186756 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:10.188005 16082 hybrid_clock.cc:648] HybridClock initialized: now 1781837110187913 us; error 56 us; skew 500 ppm
I20260619 02:45:10.188403 16082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:10 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837110, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:10.191752 16082 init.cc:377] Logged in from keytab as oryx/127.7.246.67@KRBTEST.COM (short username oryx)
I20260619 02:45:10.193260 16082 webserver.cc:460] Webserver started at http://127.7.246.67:35077/ using document root <none> and password file <none>
I20260619 02:45:10.194105 16082 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:10.194245 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:10.194595 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:10.197806 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "cb0ea263d9104e9d8869769234798319"
format_stamp: "Formatted at 2026-06-19 02:45:10 on dist-test-slave-9tkz"
server_key: "d533f5e00f15b2888142c5305c9a0e6d"
server_key_iv: "6e7688bd0e342602a3e9c611a7d9678c"
server_key_version: "encryptionkey@0"
I20260619 02:45:10.198688 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0ea263d9104e9d8869769234798319"
format_stamp: "Formatted at 2026-06-19 02:45:10 on dist-test-slave-9tkz"
server_key: "d533f5e00f15b2888142c5305c9a0e6d"
server_key_iv: "6e7688bd0e342602a3e9c611a7d9678c"
server_key_version: "encryptionkey@0"
I20260619 02:45:10.204002 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:45:10.207600 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:10.208901 16082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:45:10.209101 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "cb0ea263d9104e9d8869769234798319"
format_stamp: "Formatted at 2026-06-19 02:45:10 on dist-test-slave-9tkz"
server_key: "d533f5e00f15b2888142c5305c9a0e6d"
server_key_iv: "6e7688bd0e342602a3e9c611a7d9678c"
server_key_version: "encryptionkey@0"
I20260619 02:45:10.209343 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:10.231427 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:10.235917 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:10.236219 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:10.239130 16082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:10.239285 16082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:10.239504 16082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:10.239624 16082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:45:10 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837110, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:10.261699 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:10.239528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:53413 (local address 127.7.246.126:39729)
0619 02:45:10.239714 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:10.239720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:10.241024 (+ 1304us) server_negotiation.cc:407] Connection header received
0619 02:45:10.243207 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:10.243214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:10.243320 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:10.243466 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:10.245740 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.246489 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.248084 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.248485 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.248625 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:10.253720 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:10.253727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:10.253731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:10.253769 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:10.257612 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:10.258749 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:10.258757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:10.258764 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:10.258935 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:10.259588 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:10.259597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:10.259603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:10.260063 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:10.260248 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:10.261073 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 02:45:10.261302 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:45:10.262753 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:10.239996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.67:53413)
0619 02:45:10.240827 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:10.240871 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:10.242875 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:10.243723 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:10.243732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:10.244594 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:10.245529 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:10.245586 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.246631 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.246636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:10.247929 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:10.247940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.248722 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.248729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:10.248924 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:10.249691 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:10.249732 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:10.253465 (+ 3733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:10.257738 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:10.257748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:10.257764 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:10.258592 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:10.259102 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:10.259108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:10.259111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:10.259449 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:10.260216 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:10.260223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:10.260666 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 02:45:10.260946 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 02:45:10.261504 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":387,"threads_started":1}
I20260619 02:45:10.279738 15939 heartbeater.cc:503] Master 127.7.246.126:39729 was elected leader, sending a full tablet report...
I20260619 02:45:10.293782 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:10.278924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:44799 (local address 127.7.246.126:39729)
0619 02:45:10.279163 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:10.279172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:10.279653 (+ 481us) server_negotiation.cc:407] Connection header received
0619 02:45:10.279841 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:10.279846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:10.279918 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:10.280027 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:10.281139 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.282608 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.283905 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.284822 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.284984 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:10.287528 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:10.287537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:10.287544 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:10.287589 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:10.290862 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:10.291876 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:10.291884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:10.291888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:10.292019 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:10.292364 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:10.292370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:10.292373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:10.292641 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:10.292786 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:10.293190 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 02:45:10.293395 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:45:10.293918 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:10.278895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.7.246.67:44799)
0619 02:45:10.279547 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:10.279569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:10.279722 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:10.280049 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:10.280054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:10.280591 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:10.280979 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:10.280992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.282772 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.282777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:10.283765 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:10.283775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.284999 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.285003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:10.285132 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:10.285900 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:10.285920 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:10.287258 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:10.291042 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:10.291050 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:10.291054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:10.291754 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:10.292128 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:10.292133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:10.292137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:10.292264 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:10.292752 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:10.292757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:10.292949 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 02:45:10.293089 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:45:10.293414 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":350,"threads_started":1}
I20260619 02:45:10.305202 16082 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39363
I20260619 02:45:10.305229 16216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39363 every 8 connection(s)
I20260619 02:45:10.306442 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:10.306859 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16082
I20260619 02:45:10.307089 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:10.307519 8153 external_mini_cluster.cc:1428] Setting key ff19dfca253f98a2ab68ef1a76b02447
I20260619 02:45:10.308781 16217 heartbeater.cc:347] Connected to a master server at 127.7.246.126:39729
I20260619 02:45:10.309036 16217 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:10.309762 16217 heartbeater.cc:511] Master 127.7.246.126:39729 requested a full tablet report, sending...
I20260619 02:45:10.310946 15731 ts_manager.cc:194] Registered new tserver with Master: cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:10.311798 15731 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:53413
I20260619 02:45:10.322669 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:45:10 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:45:10 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:10 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:10.784422 16078 heartbeater.cc:503] Master 127.7.246.126:39729 was elected leader, sending a full tablet report...
I20260619 02:45:11.314121 16217 heartbeater.cc:503] Master 127.7.246.126:39729 was elected leader, sending a full tablet report...
W20260619 02:45:11.412235 16225 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.047s user 0.001s sys 0.013s
W20260619 02:45:11.413409 15827 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:10.359391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.7.246.126:39729)
0619 02:45:10.359592 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:10.359599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:10.359792 (+ 193us) server_negotiation.cc:407] Connection header received
0619 02:45:10.359952 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:10.359956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:10.360035 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:10.360122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:10.361453 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.362203 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.363454 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:10.363907 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:10.364058 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.413203 (+1049145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43312: BlockingRecv error: recv got EOF from 127.0.0.1:43312 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:45:11 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:11.430136 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.417150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.7.246.126:39729)
0619 02:45:11.417365 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.417371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.417596 (+ 225us) server_negotiation.cc:407] Connection header received
0619 02:45:11.417779 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.417783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.417858 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.417953 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:11.419053 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.419761 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.420915 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.421317 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.421403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.425013 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:11.425018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:11.425021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:11.425051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:11.427659 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:11.428260 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:11.428265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:11.428269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:11.428380 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:11.428740 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:11.428746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:11.428750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:11.429006 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:11.429310 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.429593 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.429805 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:45:11.433844 15731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:11.440645 15731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:11.468063 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.453000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.7.246.65:33641)
0619 02:45:11.453752 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.453765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.458753 (+ 4988us) server_negotiation.cc:407] Connection header received
0619 02:45:11.459054 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.459062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.459329 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.459495 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:11.460317 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.461920 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.465179 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.466690 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.466807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.467024 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.467611 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.467773 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":340,"threads_started":1}
I20260619 02:45:11.468066 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.452437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33641 (local address 127.0.0.1:40520)
0619 02:45:11.458605 (+ 6168us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:11.458651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:11.458857 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:11.459643 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:11.459651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:11.459684 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:11.460109 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.460124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.462378 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.462388 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.465018 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.465031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.467184 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.467190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.467375 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.467421 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:45:11.467553 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:45:11.467724 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5991,"thread_start_us":5857,"threads_started":1}
I20260619 02:45:11.468106 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.452387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.7.246.66:44971)
0619 02:45:11.458083 (+ 5696us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.458095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.459699 (+ 1604us) server_negotiation.cc:407] Connection header received
0619 02:45:11.459855 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.459863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.460087 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.460287 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:11.461590 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.462897 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.465567 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.466920 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.467014 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.467204 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.467704 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.467854 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5485,"thread_start_us":5352,"threads_started":1}
I20260619 02:45:11.468791 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.452157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.0.0.1:38296)
0619 02:45:11.458610 (+ 6453us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:11.458650 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:11.458820 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:11.460822 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:11.460830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:11.460865 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:11.461371 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.461379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.463044 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.463050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.465377 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.465390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.467379 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.467384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.467528 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.467562 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:45:11.468261 (+ 699us) client_negotiation.cc:240] Negotiation successful
0619 02:45:11.468417 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6154,"thread_start_us":5953,"threads_started":1}
I20260619 02:45:11.469200 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.459201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.7.246.67:39363)
0619 02:45:11.460795 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.460851 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.461471 (+ 620us) server_negotiation.cc:407] Connection header received
0619 02:45:11.461676 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.461683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.461870 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.462022 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:11.462928 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.464293 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.466805 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.467803 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.467925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.468101 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.468679 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.468876 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1314,"threads_started":1}
I20260619 02:45:11.469360 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.459123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39363 (local address 127.0.0.1:53312)
0619 02:45:11.461347 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:11.461367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:11.461483 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:11.462242 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:11.462259 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:11.462284 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:11.462743 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.462753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.464403 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.464410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.466669 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.466679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.468249 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.468257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.468482 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.468513 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:45:11.468619 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 02:45:11.468763 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"thread_start_us":1907,"threads_started":1}
I20260619 02:45:11.472129 15869 tablet_service.cc:1467] Processing CreateTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 (DEFAULT_TABLE table=test-table [id=0700a024419f4c7fa3df33e3f3b8762c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.472410 16009 tablet_service.cc:1467] Processing CreateTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 (DEFAULT_TABLE table=test-table [id=0700a024419f4c7fa3df33e3f3b8762c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.473356 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6389be0be5b34b8ab5d4960f197cf639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.473163 16147 tablet_service.cc:1467] Processing CreateTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 (DEFAULT_TABLE table=test-table [id=0700a024419f4c7fa3df33e3f3b8762c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.473732 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6389be0be5b34b8ab5d4960f197cf639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.474408 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6389be0be5b34b8ab5d4960f197cf639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.484217 16242 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Bootstrap starting.
I20260619 02:45:11.487514 16243 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Bootstrap starting.
I20260619 02:45:11.488148 16242 tablet_bootstrap.cc:654] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.489507 16242 log.cc:824] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:11.490203 16244 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Bootstrap starting.
I20260619 02:45:11.491427 16243 tablet_bootstrap.cc:654] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.492786 16244 tablet_bootstrap.cc:654] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.492931 16243 log.cc:824] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:11.493921 16244 log.cc:824] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:11.496073 16243 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: No bootstrap required, opened a new log
I20260619 02:45:11.496090 16242 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: No bootstrap required, opened a new log
I20260619 02:45:11.496351 16242 ts_tablet_manager.cc:1397] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260619 02:45:11.496366 16243 ts_tablet_manager.cc:1397] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 02:45:11.496568 16244 tablet_bootstrap.cc:492] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: No bootstrap required, opened a new log
I20260619 02:45:11.496856 16244 ts_tablet_manager.cc:1397] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260619 02:45:11.503502 16244 raft_consensus.cc:348] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.503845 16244 raft_consensus.cc:374] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.504004 16244 raft_consensus.cc:729] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0ea263d9104e9d8869769234798319, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.504573 16244 consensus_queue.cc:260] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.506263 16244 ts_tablet_manager.cc:1428] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260619 02:45:11.506971 16242 raft_consensus.cc:348] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.506973 16243 raft_consensus.cc:348] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.507371 16243 raft_consensus.cc:374] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.507439 16242 raft_consensus.cc:374] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.507520 16243 raft_consensus.cc:729] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.507568 16242 raft_consensus.cc:729] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.508159 16242 consensus_queue.cc:260] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.508158 16243 consensus_queue.cc:260] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.510097 16243 ts_tablet_manager.cc:1428] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 02:45:11.510295 16242 ts_tablet_manager.cc:1428] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 02:45:11.529421 16079 tablet.cc:2369] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:45:11.529421 15940 tablet.cc:2369] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:45:11.559602 16218 tablet.cc:2369] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:45:11.582461 16248 raft_consensus.cc:482] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:11.582835 16248 raft_consensus.cc:504] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.586930 16248 leader_election.cc:288] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971), 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641)
I20260619 02:45:11.596309 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.587952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:36627 (local address 127.7.246.65:33641)
0619 02:45:11.588174 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.588181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.588760 (+ 579us) server_negotiation.cc:407] Connection header received
0619 02:45:11.589015 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.589020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.589105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.589195 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:11.590015 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.590701 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.593410 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.594801 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.594997 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.595099 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.595882 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.596006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:45:11.596422 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.587258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.7.246.67:42253)
0619 02:45:11.588297 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:11.588376 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:11.588635 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:11.589010 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:11.589014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:11.589039 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:11.589379 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.589391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.591121 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.591127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.593374 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.593391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.595173 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.595184 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.595611 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.595715 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 02:45:11.595826 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:45:11.596007 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":129,"thread_start_us":1104,"threads_started":1}
I20260619 02:45:11.596971 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.587525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:42253 (local address 127.7.246.66:44971)
0619 02:45:11.587762 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.587772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.588551 (+ 779us) server_negotiation.cc:407] Connection header received
0619 02:45:11.588807 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.588815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.588909 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.588995 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:11.589563 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.590938 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.593595 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.594942 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.595120 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.595204 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.595854 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.596058 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:45:11.597193 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.587536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33641 (local address 127.7.246.67:36627)
0619 02:45:11.588661 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:11.588676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:11.588862 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:11.589343 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:11.589348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:11.589373 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:11.589824 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.589834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.590840 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.590847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.593255 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:11.593271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.595473 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.595481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:11.595702 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.595726 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:45:11.596533 (+ 807us) client_negotiation.cc:240] Negotiation successful
0619 02:45:11.596813 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":101,"thread_start_us":424,"threads_started":1}
I20260619 02:45:11.598119 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6389be0be5b34b8ab5d4960f197cf639" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" is_pre_election: true
I20260619 02:45:11.598502 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6389be0be5b34b8ab5d4960f197cf639" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986" is_pre_election: true
I20260619 02:45:11.598711 16033 raft_consensus.cc:2454] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 0.
I20260619 02:45:11.599066 15893 raft_consensus.cc:2454] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 0.
I20260619 02:45:11.599807 16100 leader_election.cc:302] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, cb0ea263d9104e9d8869769234798319; no voters:
I20260619 02:45:11.600250 16248 raft_consensus.cc:2781] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:11.600386 16248 raft_consensus.cc:482] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:11.600483 16248 raft_consensus.cc:3037] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:11.603469 16248 raft_consensus.cc:504] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.604195 16248 leader_election.cc:288] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 election: Requested vote from peers 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971), 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641)
I20260619 02:45:11.604655 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6389be0be5b34b8ab5d4960f197cf639" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
I20260619 02:45:11.604769 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6389be0be5b34b8ab5d4960f197cf639" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986"
I20260619 02:45:11.604871 16033 raft_consensus.cc:3037] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:11.604990 15893 raft_consensus.cc:3037] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:11.607635 16033 raft_consensus.cc:2454] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 1.
I20260619 02:45:11.608006 15893 raft_consensus.cc:2454] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 1.
I20260619 02:45:11.608168 16100 leader_election.cc:302] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, cb0ea263d9104e9d8869769234798319; no voters:
I20260619 02:45:11.608500 16248 raft_consensus.cc:2781] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:11.609011 16248 raft_consensus.cc:686] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 1 LEADER]: Becoming Leader. State: Replica: cb0ea263d9104e9d8869769234798319, State: Running, Role: LEADER
I20260619 02:45:11.609596 16248 consensus_queue.cc:237] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.614975 15731 catalog_manager.cc:5495] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0ea263d9104e9d8869769234798319 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "cb0ea263d9104e9d8869769234798319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } health_report { overall_health: HEALTHY } } }
I20260619 02:45:11.658833 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:11.653616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.7.246.67:39363)
0619 02:45:11.653794 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:11.653799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:11.653888 (+ 89us) server_negotiation.cc:407] Connection header received
0619 02:45:11.654095 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:11.654098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:11.654191 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:11.654330 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:11.654598 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.655267 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.656423 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:11.656886 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:11.657047 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:11.657521 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:11.658252 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:11.658403 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:11.658481 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:45:11.658558 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:45:11.667019 15893 raft_consensus.cc:1261] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Refusing update from remote peer cb0ea263d9104e9d8869769234798319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:11.667019 16033 raft_consensus.cc:1261] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Refusing update from remote peer cb0ea263d9104e9d8869769234798319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:11.668255 16253 consensus_queue.cc:1035] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:11.668596 16248 consensus_queue.cc:1035] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:11.680377 16256 mvcc.cc:205] Tried to move back new op lower bound from 7298404809378078720 to 7298404809159000064. Current Snapshot: MvccSnapshot[applied={T|T < 7298404809378078720}]
I20260619 02:45:11.683355 16257 mvcc.cc:205] Tried to move back new op lower bound from 7298404809378078720 to 7298404809159000064. Current Snapshot: MvccSnapshot[applied={T|T < 7298404809378078720}]
I20260619 02:45:11.692001 15732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:11.692229 15732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:11.695731 15732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Sending DeleteTablet for 3 replicas of tablet 6389be0be5b34b8ab5d4960f197cf639
I20260619 02:45:11.696863 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:11 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260619 02:45:11.697597 15869 tablet_service.cc:1514] Processing DeleteTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:11 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:40520
I20260619 02:45:11.697631 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 6389be0be5b34b8ab5d4960f197cf639 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:11 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260619 02:45:11.697808 16265 tablet_replica.cc:330] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: stopping tablet replica
I20260619 02:45:11.698212 16265 raft_consensus.cc:2229] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:11.698719 16265 raft_consensus.cc:2258] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:11.698618 15731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:11.699244 15731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:11.700803 16265 ts_tablet_manager.cc:1905] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:11.707957 16146 tablet_service.cc:1467] Processing CreateTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 (DEFAULT_TABLE table=test-table [id=39dd9d2964d44f859453a242540be6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.707962 15868 tablet_service.cc:1467] Processing CreateTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 (DEFAULT_TABLE table=test-table [id=39dd9d2964d44f859453a242540be6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.708490 16146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02c1c1b8e3e481d963ac3082ec44394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.708513 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02c1c1b8e3e481d963ac3082ec44394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.709054 16265 ts_tablet_manager.cc:1918] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:11.709194 16265 log.cc:1196] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/6389be0be5b34b8ab5d4960f197cf639
I20260619 02:45:11.709730 16265 ts_tablet_manager.cc:1939] T 6389be0be5b34b8ab5d4960f197cf639 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting consensus metadata
I20260619 02:45:11.712904 15718 catalog_manager.cc:4841] TS 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971): tablet 6389be0be5b34b8ab5d4960f197cf639 (table test-table [id=0700a024419f4c7fa3df33e3f3b8762c]) successfully deleted
I20260619 02:45:11.714079 16009 tablet_service.cc:1467] Processing CreateTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 (DEFAULT_TABLE table=test-table [id=39dd9d2964d44f859453a242540be6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:11.714497 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02c1c1b8e3e481d963ac3082ec44394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:11.719143 16267 tablet_replica.cc:330] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: stopping tablet replica
I20260619 02:45:11.719311 16242 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Bootstrap starting.
I20260619 02:45:11.719574 16267 raft_consensus.cc:2229] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:11.720178 16267 raft_consensus.cc:2258] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:11.720968 16242 tablet_bootstrap.cc:654] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.720990 16266 tablet_replica.cc:330] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: stopping tablet replica
I20260619 02:45:11.721532 16244 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Bootstrap starting.
I20260619 02:45:11.721671 16266 raft_consensus.cc:2229] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:11.722121 16243 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Bootstrap starting.
I20260619 02:45:11.722110 16266 raft_consensus.cc:2258] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:11.723248 16244 tablet_bootstrap.cc:654] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.723623 16243 tablet_bootstrap.cc:654] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:11.724299 16267 ts_tablet_manager.cc:1905] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:11.731513 16266 ts_tablet_manager.cc:1905] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:11.737150 16242 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: No bootstrap required, opened a new log
I20260619 02:45:11.737376 16242 ts_tablet_manager.cc:1397] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s
I20260619 02:45:11.737512 16243 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: No bootstrap required, opened a new log
I20260619 02:45:11.737681 16243 ts_tablet_manager.cc:1397] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20260619 02:45:11.738729 16243 raft_consensus.cc:348] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.739014 16243 raft_consensus.cc:374] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.739113 16243 raft_consensus.cc:729] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.739174 16242 raft_consensus.cc:348] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.739418 16242 raft_consensus.cc:374] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.739544 16242 raft_consensus.cc:729] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.739915 16242 consensus_queue.cc:260] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.740000 16243 consensus_queue.cc:260] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.740733 16243 ts_tablet_manager.cc:1428] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:45:11.742605 16244 tablet_bootstrap.cc:492] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: No bootstrap required, opened a new log
I20260619 02:45:11.742926 16244 ts_tablet_manager.cc:1397] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.002s
I20260619 02:45:11.743844 16244 raft_consensus.cc:348] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.744133 16244 raft_consensus.cc:374] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:11.744282 16244 raft_consensus.cc:729] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0ea263d9104e9d8869769234798319, State: Initialized, Role: FOLLOWER
I20260619 02:45:11.744658 16244 consensus_queue.cc:260] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:11.746687 16242 ts_tablet_manager.cc:1428] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260619 02:45:11.747259 16244 ts_tablet_manager.cc:1428] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 02:45:11.749215 16267 ts_tablet_manager.cc:1918] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:11.749393 16267 log.cc:1196] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/6389be0be5b34b8ab5d4960f197cf639
I20260619 02:45:11.750020 16267 ts_tablet_manager.cc:1939] T 6389be0be5b34b8ab5d4960f197cf639 P cb0ea263d9104e9d8869769234798319: Deleting consensus metadata
I20260619 02:45:11.750312 16266 ts_tablet_manager.cc:1918] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:11.750465 16266 log.cc:1196] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/6389be0be5b34b8ab5d4960f197cf639
I20260619 02:45:11.750955 16266 ts_tablet_manager.cc:1939] T 6389be0be5b34b8ab5d4960f197cf639 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting consensus metadata
I20260619 02:45:11.751758 15717 catalog_manager.cc:4841] TS cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363): tablet 6389be0be5b34b8ab5d4960f197cf639 (table test-table [id=0700a024419f4c7fa3df33e3f3b8762c]) successfully deleted
I20260619 02:45:11.752496 15717 catalog_manager.cc:4841] TS 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641): tablet 6389be0be5b34b8ab5d4960f197cf639 (table test-table [id=0700a024419f4c7fa3df33e3f3b8762c]) successfully deleted
I20260619 02:45:12.137140 16249 raft_consensus.cc:482] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:12.137466 16249 raft_consensus.cc:504] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.139011 16249 leader_election.cc:288] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:12.145709 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.139442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:50849 (local address 127.7.246.67:39363)
0619 02:45:12.139641 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.139650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.140385 (+ 735us) server_negotiation.cc:407] Connection header received
0619 02:45:12.140595 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.140600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.140667 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.140757 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:12.141327 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.142154 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.144019 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.144740 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.144902 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.144975 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.145294 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.145414 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:45:12.145817 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.139275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39363 (local address 127.7.246.65:50849)
0619 02:45:12.140256 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.140274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.140423 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.140771 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.140775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.140797 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:12.141189 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.141196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.142282 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.142292 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.143868 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.143879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.144902 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.144908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.145100 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.145122 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.145266 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.145408 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":386,"thread_start_us":296,"threads_started":1}
I20260619 02:45:12.145994 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.139464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:33177 (local address 127.7.246.66:44971)
0619 02:45:12.140010 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.140019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.140041 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:45:12.140309 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.140315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.140412 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.140547 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:12.141164 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.142094 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.144043 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.144874 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.145080 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.145164 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.145468 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.145635 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":285,"threads_started":1}
I20260619 02:45:12.146211 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.139275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.7.246.65:33177)
0619 02:45:12.139948 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.139969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.140147 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.140565 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.140570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.140598 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:12.140953 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.140962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.142231 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.142239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.143878 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.143890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.145119 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.145126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.145302 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.145327 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.145444 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.145593 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":1,"thread_start_us":332,"threads_started":1}
I20260619 02:45:12.146920 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02c1c1b8e3e481d963ac3082ec44394" candidate_uuid: "8338ba0738f94e768c4d40a8bc6dc986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" is_pre_election: true
I20260619 02:45:12.146909 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02c1c1b8e3e481d963ac3082ec44394" candidate_uuid: "8338ba0738f94e768c4d40a8bc6dc986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319" is_pre_election: true
I20260619 02:45:12.147159 16033 raft_consensus.cc:2454] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8338ba0738f94e768c4d40a8bc6dc986 in term 0.
I20260619 02:45:12.147282 16172 raft_consensus.cc:2454] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8338ba0738f94e768c4d40a8bc6dc986 in term 0.
I20260619 02:45:12.147625 15821 leader_election.cc:302] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, 8338ba0738f94e768c4d40a8bc6dc986; no voters:
I20260619 02:45:12.148046 16249 raft_consensus.cc:2781] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:12.148147 16249 raft_consensus.cc:482] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:12.148272 16249 raft_consensus.cc:3037] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.150496 16249 raft_consensus.cc:504] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.151042 16249 leader_election.cc:288] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [CANDIDATE]: Term 1 election: Requested vote from peers 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:12.151559 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02c1c1b8e3e481d963ac3082ec44394" candidate_uuid: "8338ba0738f94e768c4d40a8bc6dc986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319"
I20260619 02:45:12.151597 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02c1c1b8e3e481d963ac3082ec44394" candidate_uuid: "8338ba0738f94e768c4d40a8bc6dc986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
I20260619 02:45:12.151767 16172 raft_consensus.cc:3037] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.151767 16033 raft_consensus.cc:3037] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.154681 16033 raft_consensus.cc:2454] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8338ba0738f94e768c4d40a8bc6dc986 in term 1.
I20260619 02:45:12.154681 16172 raft_consensus.cc:2454] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8338ba0738f94e768c4d40a8bc6dc986 in term 1.
I20260619 02:45:12.155078 15821 leader_election.cc:302] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, 8338ba0738f94e768c4d40a8bc6dc986; no voters:
I20260619 02:45:12.155377 16249 raft_consensus.cc:2781] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:12.155786 16249 raft_consensus.cc:686] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 LEADER]: Becoming Leader. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Running, Role: LEADER
I20260619 02:45:12.156139 16249 consensus_queue.cc:237] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.159484 15731 catalog_manager.cc:5495] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65). New cstate: current_term: 1 leader_uuid: "8338ba0738f94e768c4d40a8bc6dc986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:45:12 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837108, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.126@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:12.210518 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.191687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39729 (local address 127.0.0.1:43330)
0619 02:45:12.192504 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.192518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.192685 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.193012 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.193016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.193784 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:12.193958 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.193967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.195056 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.195062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.196185 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.196198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.196916 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.196921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.197135 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.203392 (+ 6257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:12.203410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:12.205691 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:12.208384 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:12.208394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:12.208409 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:12.208786 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:12.209089 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:12.209092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:12.209094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:12.209217 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:12.209713 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:12.209719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:12.209850 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.209990 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.210208 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":453,"threads_started":1}
I20260619 02:45:12.210654 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.191676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.7.246.126:39729)
0619 02:45:12.192437 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.192443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.192608 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:45:12.192796 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.192800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.192865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.192993 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:12.194087 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.194912 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.196315 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.196764 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.196898 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.205899 (+ 9001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:12.205904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:12.205907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:12.205946 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:12.208238 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:12.208875 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:12.208879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:12.208881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:12.209001 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:12.209298 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:12.209301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:12.209303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:12.209523 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:12.209711 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.210090 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.210305 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":609,"thread_start_us":433,"threads_started":1}
I20260619 02:45:12.215400 15731 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:43330:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:45:12.216457 15731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:12.222980 15868 tablet_service.cc:1467] Processing CreateTablet for tablet d464db0cfedd4b3da367b6d03de1098c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85304b781b84752b0c6db48dee64491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:45:12.223021 16009 tablet_service.cc:1467] Processing CreateTablet for tablet d464db0cfedd4b3da367b6d03de1098c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85304b781b84752b0c6db48dee64491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:45:12.223345 16147 tablet_service.cc:1467] Processing CreateTablet for tablet d464db0cfedd4b3da367b6d03de1098c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a85304b781b84752b0c6db48dee64491]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:45:12.223515 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d464db0cfedd4b3da367b6d03de1098c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.223519 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d464db0cfedd4b3da367b6d03de1098c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.223815 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d464db0cfedd4b3da367b6d03de1098c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.229468 16243 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: Bootstrap starting.
I20260619 02:45:12.230470 16242 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: Bootstrap starting.
I20260619 02:45:12.230479 16244 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: Bootstrap starting.
I20260619 02:45:12.231087 16243 tablet_bootstrap.cc:654] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.232174 16244 tablet_bootstrap.cc:654] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.232235 16242 tablet_bootstrap.cc:654] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.234009 16243 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: No bootstrap required, opened a new log
I20260619 02:45:12.234201 16243 ts_tablet_manager.cc:1397] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 02:45:12.234970 16243 raft_consensus.cc:348] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.235180 16243 raft_consensus.cc:374] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.235276 16244 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: No bootstrap required, opened a new log
I20260619 02:45:12.235284 16243 raft_consensus.cc:729] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.235440 16244 ts_tablet_manager.cc:1397] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 02:45:12.235451 16242 tablet_bootstrap.cc:492] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: No bootstrap required, opened a new log
I20260619 02:45:12.235581 16243 consensus_queue.cc:260] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.235773 16242 ts_tablet_manager.cc:1397] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 02:45:12.236168 16243 ts_tablet_manager.cc:1428] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:45:12.236162 16244 raft_consensus.cc:348] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.236377 16244 raft_consensus.cc:374] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.236450 16244 raft_consensus.cc:729] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0ea263d9104e9d8869769234798319, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.236471 16242 raft_consensus.cc:348] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.236680 16242 raft_consensus.cc:374] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.236748 16242 raft_consensus.cc:729] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.236681 16244 consensus_queue.cc:260] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.236994 16242 consensus_queue.cc:260] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.236994 16250 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.237385 16250 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.237372 16253 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.237599 16242 ts_tablet_manager.cc:1428] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:45:12.237481 16249 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.237684 16253 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.238615 16244 ts_tablet_manager.cc:1428] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:12.239218 16249 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.243829 16250 raft_consensus.cc:482] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:12.243995 16250 raft_consensus.cc:504] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.245371 16250 leader_election.cc:288] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:12.253377 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.245656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44827 (local address 127.7.246.65:33641)
0619 02:45:12.246394 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.246402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.246422 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:45:12.246499 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.246507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.246604 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.246718 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:12.247456 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.248696 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.251223 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.252371 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.252577 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.252664 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.252977 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.253073 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":304,"threads_started":1}
I20260619 02:45:12.253572 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.245599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33641 (local address 127.7.246.66:44827)
0619 02:45:12.246073 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.246099 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.246235 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.246770 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.246775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.246802 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:12.247175 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.247190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.248874 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.248878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.251079 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.251093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.252595 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.252604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.252763 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.252828 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.252940 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.253085 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":24,"thread_start_us":198,"threads_started":1}
I20260619 02:45:12.253686 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.246310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:58529 (local address 127.7.246.67:39363)
0619 02:45:12.246467 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.246472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.247658 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 02:45:12.247801 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.247804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.247861 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.247928 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:12.248470 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.249235 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.251483 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.252716 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.252830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.252875 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.253257 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.253418 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:45:12.253741 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.246187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39363 (local address 127.7.246.66:58529)
0619 02:45:12.247558 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.247575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.247691 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.247966 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.247970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.247994 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:12.248330 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.248340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.249355 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.249360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.251354 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.251363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.252929 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.252936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.253106 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.253134 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.253231 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.253341 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"mutex_wait_us":1,"thread_start_us":1034,"threads_started":1}
I20260619 02:45:12.254300 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d464db0cfedd4b3da367b6d03de1098c" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986" is_pre_election: true
I20260619 02:45:12.254552 15893 raft_consensus.cc:2454] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 0.
I20260619 02:45:12.254783 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d464db0cfedd4b3da367b6d03de1098c" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319" is_pre_election: true
I20260619 02:45:12.255044 16172 raft_consensus.cc:2454] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 0.
I20260619 02:45:12.255105 15960 leader_election.cc:302] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, 8338ba0738f94e768c4d40a8bc6dc986; no voters:
I20260619 02:45:12.255549 16250 raft_consensus.cc:2781] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:12.255668 16250 raft_consensus.cc:482] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:12.255786 16250 raft_consensus.cc:3037] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.257835 16250 raft_consensus.cc:504] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.258378 16250 leader_election.cc:288] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 election: Requested vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:12.258878 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d464db0cfedd4b3da367b6d03de1098c" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986"
I20260619 02:45:12.258908 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d464db0cfedd4b3da367b6d03de1098c" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319"
I20260619 02:45:12.259086 15893 raft_consensus.cc:3037] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.259128 16172 raft_consensus.cc:3037] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.261150 15893 raft_consensus.cc:2454] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 1.
I20260619 02:45:12.261237 16172 raft_consensus.cc:2454] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 1.
I20260619 02:45:12.261652 15960 leader_election.cc:302] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, 8338ba0738f94e768c4d40a8bc6dc986; no voters:
I20260619 02:45:12.261891 16250 raft_consensus.cc:2781] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:12.262303 16250 raft_consensus.cc:686] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 LEADER]: Becoming Leader. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Running, Role: LEADER
I20260619 02:45:12.262741 16250 consensus_queue.cc:237] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.264920 16289 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: TxnStatusTablet state changed. Reason: New leader 7c8ae65d3bbe4304832088cc3cc42e8f. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.265388 16289 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:45:12.266116 16291 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:45:12.266326 16291 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:45:12.266621 15732 catalog_manager.cc:5495] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66). New cstate: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } health_report { overall_health: UNKNOWN } } }
I20260619 02:45:12.286572 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.280885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.7.246.66:44971)
0619 02:45:12.281074 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.281080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.281637 (+ 557us) server_negotiation.cc:407] Connection header received
0619 02:45:12.281737 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.281741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.281824 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.281971 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:12.282316 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.283043 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.284327 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.284768 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.284903 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.285358 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.285987 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.286130 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.286207 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.286284 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:45:12.286624 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.280900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.0.0.1:38300)
0619 02:45:12.281261 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.281281 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.281422 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.281985 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.281990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.282002 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:45:12.282179 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.282187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.283180 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.283188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.284176 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.284186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.284916 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.284922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.285110 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.285169 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.286089 (+ 920us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.286110 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.286192 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.286337 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 02:45:12.291060 15893 raft_consensus.cc:1261] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Refusing update from remote peer 7c8ae65d3bbe4304832088cc3cc42e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.291395 16172 raft_consensus.cc:1261] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Refusing update from remote peer 7c8ae65d3bbe4304832088cc3cc42e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.291816 16289 consensus_queue.cc:1035] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.292095 16250 consensus_queue.cc:1035] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.305593 16253 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: TxnStatusTablet state changed. Reason: New leader 7c8ae65d3bbe4304832088cc3cc42e8f. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.305881 16253 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.318135 16289 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.318440 16289 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:45:12.322005 16253 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.322260 16253 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P cb0ea263d9104e9d8869769234798319: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.323643 16249 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: TxnStatusTablet state changed. Reason: New leader 7c8ae65d3bbe4304832088cc3cc42e8f. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.323992 16249 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.326368 16250 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.326694 16250 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 7c8ae65d3bbe4304832088cc3cc42e8f: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:45:12.353780 16249 tablet_replica.cc:439] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } } }
I20260619 02:45:12.354115 16249 tablet_replica.cc:442] T d464db0cfedd4b3da367b6d03de1098c P 8338ba0738f94e768c4d40a8bc6dc986: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:45:12.358731 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.341762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.7.246.65:33641)
0619 02:45:12.342015 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.342024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.351591 (+ 9567us) server_negotiation.cc:407] Connection header received
0619 02:45:12.352222 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.352231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.352376 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.352567 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:12.353311 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.354459 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.356496 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.356887 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.356981 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.357484 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.358130 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.358236 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.358385 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.358462 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:45:12.376044 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.369762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:39445 (local address 127.7.246.66:44971)
0619 02:45:12.369978 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.369988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.370596 (+ 608us) server_negotiation.cc:407] Connection header received
0619 02:45:12.370799 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.370803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.370867 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.371147 (+ 280us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:12.371496 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.372259 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.374294 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.374720 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.374820 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.375335 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.375459 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.375547 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.375688 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.375748 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:45:12.376199 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.369726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.7.246.65:39445)
0619 02:45:12.370490 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.370511 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.370670 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.370987 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.370992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.371006 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:45:12.371335 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.371346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.372385 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.372390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.374163 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.374179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.374906 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.374912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.375098 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.375174 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.375575 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.375586 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.375666 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.375807 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":420,"threads_started":1}
I20260619 02:45:12.386744 16172 raft_consensus.cc:1261] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Refusing update from remote peer 8338ba0738f94e768c4d40a8bc6dc986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.386752 16033 raft_consensus.cc:1261] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Refusing update from remote peer 8338ba0738f94e768c4d40a8bc6dc986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.387420 16274 consensus_queue.cc:1035] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.387763 16249 consensus_queue.cc:1035] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.393460 16295 mvcc.cc:205] Tried to move back new op lower bound from 7298404812329791488 to 7298404811394924544. Current Snapshot: MvccSnapshot[applied={T|T < 7298404812329791488}]
I20260619 02:45:12.429842 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.423115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:54093 (local address 127.7.246.65:33641)
0619 02:45:12.423272 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.423276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.423931 (+ 655us) server_negotiation.cc:407] Connection header received
0619 02:45:12.424136 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.424141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.424291 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.424370 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:12.425245 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.426173 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.427783 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.428383 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.428509 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.429158 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.429269 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.429354 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.429504 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.429617 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:45:12.430109 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.423082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33641 (local address 127.7.246.66:54093)
0619 02:45:12.423825 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:12.423845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:12.424001 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:12.424559 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:12.424564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:12.424576 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:45:12.425024 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.425039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.426286 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.426293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.427644 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:12.427656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.428690 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.428698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:12.428923 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.428994 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.429385 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.429397 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:45:12.429494 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:45:12.429694 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":411,"threads_started":1}
I20260619 02:45:12.463955 15730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:12.464228 15730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:12.468102 15730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Sending DeleteTablet for 3 replicas of tablet f02c1c1b8e3e481d963ac3082ec44394
I20260619 02:45:12.469214 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260619 02:45:12.469269 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260619 02:45:12.469987 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet f02c1c1b8e3e481d963ac3082ec44394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:40520
I20260619 02:45:12.474005 16319 tablet_replica.cc:330] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: stopping tablet replica
I20260619 02:45:12.475733 16318 tablet_replica.cc:330] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: stopping tablet replica
I20260619 02:45:12.476045 16319 raft_consensus.cc:2229] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:12.476126 16318 raft_consensus.cc:2229] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:12.476778 16319 raft_consensus.cc:2258] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.476589 16318 raft_consensus.cc:2258] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.478107 16319 ts_tablet_manager.cc:1905] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.479053 16318 ts_tablet_manager.cc:1905] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.482400 16317 tablet_replica.cc:330] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: stopping tablet replica
I20260619 02:45:12.482758 16317 raft_consensus.cc:2229] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:12.483062 16317 raft_consensus.cc:2258] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.484226 16317 ts_tablet_manager.cc:1905] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.491544 16317 ts_tablet_manager.cc:1918] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:12.494081 16319 ts_tablet_manager.cc:1918] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:12.494323 16319 log.cc:1196] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/f02c1c1b8e3e481d963ac3082ec44394
I20260619 02:45:12.494835 16319 ts_tablet_manager.cc:1939] T f02c1c1b8e3e481d963ac3082ec44394 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting consensus metadata
I20260619 02:45:12.496362 15717 catalog_manager.cc:4841] TS 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641): tablet f02c1c1b8e3e481d963ac3082ec44394 (table test-table [id=39dd9d2964d44f859453a242540be6b8]) successfully deleted
I20260619 02:45:12.497562 16318 ts_tablet_manager.cc:1918] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:12.497865 16318 log.cc:1196] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/f02c1c1b8e3e481d963ac3082ec44394
I20260619 02:45:12.498436 16318 ts_tablet_manager.cc:1939] T f02c1c1b8e3e481d963ac3082ec44394 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting consensus metadata
I20260619 02:45:12.499599 15718 catalog_manager.cc:4841] TS 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971): tablet f02c1c1b8e3e481d963ac3082ec44394 (table test-table [id=39dd9d2964d44f859453a242540be6b8]) successfully deleted
I20260619 02:45:12.500010 16317 log.cc:1196] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/f02c1c1b8e3e481d963ac3082ec44394
I20260619 02:45:12.500473 16317 ts_tablet_manager.cc:1939] T f02c1c1b8e3e481d963ac3082ec44394 P cb0ea263d9104e9d8869769234798319: Deleting consensus metadata
I20260619 02:45:12.501585 15717 catalog_manager.cc:4841] TS cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363): tablet f02c1c1b8e3e481d963ac3082ec44394 (table test-table [id=39dd9d2964d44f859453a242540be6b8]) successfully deleted
Jun 19 02:45:12 dist-test-slave-9tkz krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:45:12 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:12 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:45:12.538005 16281 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:12.526907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.7.246.126:39729)
0619 02:45:12.527114 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.527123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.527692 (+ 569us) server_negotiation.cc:407] Connection header received
0619 02:45:12.527920 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.527929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.527993 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.528096 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:12.529349 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.530182 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.531489 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.532065 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.532219 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.537719 (+ 5500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43346: BlockingRecv error: recv got EOF from 127.0.0.1:43346 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:45:12 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM
I20260619 02:45:12.555907 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.541753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.7.246.126:39729)
0619 02:45:12.541943 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.541949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.542640 (+ 691us) server_negotiation.cc:407] Connection header received
0619 02:45:12.542920 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.542926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.543017 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.543072 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:12.544298 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.544997 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.546281 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.546698 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.546832 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.550559 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:12.550565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:12.550568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:12.550599 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:12.553175 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:12.553941 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:12.553945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:12.553948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:12.554060 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:12.554481 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:12.554485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:12.554487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:12.554738 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:12.555009 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.555311 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.555547 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:45:12.559614 15730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:12.560391 15730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:12.567446 15868 tablet_service.cc:1467] Processing CreateTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (DEFAULT_TABLE table=test-table [id=cb2a5cd10db442dc9af41d6b14046e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.567802 16009 tablet_service.cc:1467] Processing CreateTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (DEFAULT_TABLE table=test-table [id=cb2a5cd10db442dc9af41d6b14046e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.567930 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b97f9b76de949f1bb9bba6bcf04ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.568210 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b97f9b76de949f1bb9bba6bcf04ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.572634 16242 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Bootstrap starting.
I20260619 02:45:12.573679 16147 tablet_service.cc:1467] Processing CreateTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (DEFAULT_TABLE table=test-table [id=cb2a5cd10db442dc9af41d6b14046e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.574066 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b97f9b76de949f1bb9bba6bcf04ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.574287 16242 tablet_bootstrap.cc:654] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.574442 16243 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Bootstrap starting.
I20260619 02:45:12.576099 16243 tablet_bootstrap.cc:654] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.588590 16243 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: No bootstrap required, opened a new log
I20260619 02:45:12.588843 16243 ts_tablet_manager.cc:1397] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20260619 02:45:12.589775 16243 raft_consensus.cc:348] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.590194 16243 raft_consensus.cc:374] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.590348 16243 raft_consensus.cc:729] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.590399 16244 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Bootstrap starting.
I20260619 02:45:12.590725 16243 consensus_queue.cc:260] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.592585 16244 tablet_bootstrap.cc:654] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.593832 16243 ts_tablet_manager.cc:1428] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:45:12.607044 16244 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: No bootstrap required, opened a new log
I20260619 02:45:12.607367 16244 ts_tablet_manager.cc:1397] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260619 02:45:12.608484 16244 raft_consensus.cc:348] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.608932 16244 raft_consensus.cc:374] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.609117 16244 raft_consensus.cc:729] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0ea263d9104e9d8869769234798319, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.610904 16244 consensus_queue.cc:260] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.611702 16244 ts_tablet_manager.cc:1428] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 02:45:12.614714 16242 tablet_bootstrap.cc:492] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: No bootstrap required, opened a new log
I20260619 02:45:12.614964 16242 ts_tablet_manager.cc:1397] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.021s
I20260619 02:45:12.615887 16242 raft_consensus.cc:348] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.616267 16242 raft_consensus.cc:374] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.616428 16242 raft_consensus.cc:729] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.616689 16242 consensus_queue.cc:260] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.617264 16242 ts_tablet_manager.cc:1428] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:45:12.820232 16253 raft_consensus.cc:482] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:12.820472 16253 raft_consensus.cc:504] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.821105 16253 leader_election.cc:288] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971)
I20260619 02:45:12.821717 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b97f9b76de949f1bb9bba6bcf04ada8" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986" is_pre_election: true
I20260619 02:45:12.821784 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b97f9b76de949f1bb9bba6bcf04ada8" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" is_pre_election: true
I20260619 02:45:12.822001 16033 raft_consensus.cc:2454] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 0.
I20260619 02:45:12.822013 15893 raft_consensus.cc:2454] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 0.
I20260619 02:45:12.822491 16099 leader_election.cc:302] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8338ba0738f94e768c4d40a8bc6dc986, cb0ea263d9104e9d8869769234798319; no voters:
I20260619 02:45:12.822829 16253 raft_consensus.cc:2781] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:12.822945 16253 raft_consensus.cc:482] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:12.823051 16253 raft_consensus.cc:3037] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.826066 16253 raft_consensus.cc:504] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.826741 16253 leader_election.cc:288] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 election: Requested vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971)
I20260619 02:45:12.827255 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b97f9b76de949f1bb9bba6bcf04ada8" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986"
I20260619 02:45:12.827266 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b97f9b76de949f1bb9bba6bcf04ada8" candidate_uuid: "cb0ea263d9104e9d8869769234798319" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f"
I20260619 02:45:12.827463 16033 raft_consensus.cc:3037] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.827482 15893 raft_consensus.cc:3037] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:12.830359 16033 raft_consensus.cc:2454] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 1.
I20260619 02:45:12.830569 15893 raft_consensus.cc:2454] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0ea263d9104e9d8869769234798319 in term 1.
I20260619 02:45:12.830776 16100 leader_election.cc:302] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, cb0ea263d9104e9d8869769234798319; no voters:
I20260619 02:45:12.831125 16253 raft_consensus.cc:2781] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:12.831506 16253 raft_consensus.cc:686] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 1 LEADER]: Becoming Leader. State: Replica: cb0ea263d9104e9d8869769234798319, State: Running, Role: LEADER
I20260619 02:45:12.831844 16253 consensus_queue.cc:237] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.834561 15730 catalog_manager.cc:5495] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0ea263d9104e9d8869769234798319 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "cb0ea263d9104e9d8869769234798319" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } health_report { overall_health: HEALTHY } } }
I20260619 02:45:12.903272 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:12.897419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.7.246.67:39363)
0619 02:45:12.898043 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:12.898047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:12.898067 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:45:12.898122 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:12.898126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:12.898184 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:12.898262 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:12.898701 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.899564 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.901429 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:12.901903 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:12.902001 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:12.902517 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:12.902623 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:12.902688 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:12.902881 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:45:12.902947 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":346,"threads_started":1}
I20260619 02:45:12.906030 16033 raft_consensus.cc:1261] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Refusing update from remote peer cb0ea263d9104e9d8869769234798319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.906092 15893 raft_consensus.cc:1261] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Refusing update from remote peer cb0ea263d9104e9d8869769234798319: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:12.906636 16253 consensus_queue.cc:1035] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.906949 16335 consensus_queue.cc:1035] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:12.923702 15730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:12.924037 15730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:12.927871 15730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Sending DeleteTablet for 3 replicas of tablet 0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.928956 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:40520
I20260619 02:45:12.929080 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260619 02:45:12.929347 16319 tablet_replica.cc:330] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: stopping tablet replica
I20260619 02:45:12.929349 16318 tablet_replica.cc:330] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: stopping tablet replica
I20260619 02:45:12.929515 16319 raft_consensus.cc:2229] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:12.929601 16318 raft_consensus.cc:2229] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:12.929752 16318 pending_rounds.cc:62] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Trying to abort 2 pending ops.
I20260619 02:45:12.929754 16319 raft_consensus.cc:2258] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.929886 16318 pending_rounds.cc:68] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298404814457589760 op_type: WRITE_OP write_request { tablet_id: "0b97f9b76de949f1bb9bba6bcf04ada8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\344\334\322\321\006\"8\n\ntest-admin\022*\n cb2a5cd10db442dc9af41d6b14046e98\020\001\030\001 \001(\001" signature: "<redacted>""j@]!-\253\006\027\202H\343A\303|\350\251m\262%&\352\214M\027_\261\003\336\257\212\370!X\n\256\025\315.n\203\242\377<&\327#\206\263$4l\233\277+\371\314\315\034\370\202\214\303\031W" signing_key_seq_num: 0 } } request_id { client_id: "5f382a77b0ff4c5499001b79041522ee" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 02:45:12.930290 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260619 02:45:12.930367 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260619 02:45:12.930622 16317 tablet_replica.cc:330] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: stopping tablet replica
I20260619 02:45:12.930608 16318 pending_rounds.cc:68] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298404814161924096 op_type: NO_OP noop_request { }
I20260619 02:45:12.930751 16318 raft_consensus.cc:2866] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 02:45:12.930799 16317 raft_consensus.cc:2229] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:12.930886 16318 raft_consensus.cc:2258] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.931108 16317 raft_consensus.cc:2258] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:12.931702 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:12 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260619 02:45:12.932075 16318 ts_tablet_manager.cc:1905] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.932495 16317 ts_tablet_manager.cc:1905] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.933346 15730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:12.934017 15730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:12.936462 16319 ts_tablet_manager.cc:1905] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:12.938596 16318 ts_tablet_manager.cc:1918] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:12.938820 16318 log.cc:1196] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.939471 16318 ts_tablet_manager.cc:1939] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting consensus metadata
I20260619 02:45:12.941062 15718 catalog_manager.cc:4841] TS 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971): tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (table test-table [id=cb2a5cd10db442dc9af41d6b14046e98]) successfully deleted
I20260619 02:45:12.941152 16317 ts_tablet_manager.cc:1918] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:12.941289 16317 log.cc:1196] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.941792 16317 ts_tablet_manager.cc:1939] T 0b97f9b76de949f1bb9bba6bcf04ada8 P cb0ea263d9104e9d8869769234798319: Deleting consensus metadata
W20260619 02:45:12.941803 15718 catalog_manager.cc:4808] TS 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971): delete failed for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 because the tablet was not found. No further retry: Not found: Tablet not found: 0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.942691 16009 tablet_service.cc:1467] Processing CreateTablet for tablet cdd5643da99c428abfaaa45918a378d8 (DEFAULT_TABLE table=test-table [id=939f64cd8369449c980deaae5d4f13a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.943077 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd5643da99c428abfaaa45918a378d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.943341 15868 tablet_service.cc:1467] Processing CreateTablet for tablet cdd5643da99c428abfaaa45918a378d8 (DEFAULT_TABLE table=test-table [id=939f64cd8369449c980deaae5d4f13a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.943697 15868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd5643da99c428abfaaa45918a378d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.944118 16319 ts_tablet_manager.cc:1918] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:12.944253 16319 log.cc:1196] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.944576 16319 ts_tablet_manager.cc:1939] T 0b97f9b76de949f1bb9bba6bcf04ada8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting consensus metadata
I20260619 02:45:12.949400 16243 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Bootstrap starting.
I20260619 02:45:12.949772 16147 tablet_service.cc:1467] Processing CreateTablet for tablet cdd5643da99c428abfaaa45918a378d8 (DEFAULT_TABLE table=test-table [id=939f64cd8369449c980deaae5d4f13a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:12.950161 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd5643da99c428abfaaa45918a378d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:12.951016 16243 tablet_bootstrap.cc:654] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.957954 15717 catalog_manager.cc:4841] TS 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641): tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (table test-table [id=cb2a5cd10db442dc9af41d6b14046e98]) successfully deleted
I20260619 02:45:12.958204 15717 catalog_manager.cc:4841] TS cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363): tablet 0b97f9b76de949f1bb9bba6bcf04ada8 (table test-table [id=cb2a5cd10db442dc9af41d6b14046e98]) successfully deleted
W20260619 02:45:12.958436 15717 catalog_manager.cc:4808] TS cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363): delete failed for tablet 0b97f9b76de949f1bb9bba6bcf04ada8 because the tablet was not found. No further retry: Not found: Tablet not found: 0b97f9b76de949f1bb9bba6bcf04ada8
I20260619 02:45:12.959014 16242 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Bootstrap starting.
I20260619 02:45:12.959436 16243 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: No bootstrap required, opened a new log
I20260619 02:45:12.959688 16243 ts_tablet_manager.cc:1397] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20260619 02:45:12.960032 16244 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Bootstrap starting.
I20260619 02:45:12.960558 16243 raft_consensus.cc:348] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.960826 16242 tablet_bootstrap.cc:654] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.960868 16243 raft_consensus.cc:374] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.960947 16243 raft_consensus.cc:729] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.961186 16243 consensus_queue.cc:260] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.961709 16244 tablet_bootstrap.cc:654] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:12.961843 16243 ts_tablet_manager.cc:1428] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:45:12.964526 16244 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: No bootstrap required, opened a new log
I20260619 02:45:12.964522 16242 tablet_bootstrap.cc:492] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: No bootstrap required, opened a new log
I20260619 02:45:12.964721 16242 ts_tablet_manager.cc:1397] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:45:12.964792 16244 ts_tablet_manager.cc:1397] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 02:45:12.965662 16242 raft_consensus.cc:348] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.965983 16242 raft_consensus.cc:374] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.966105 16242 raft_consensus.cc:729] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8338ba0738f94e768c4d40a8bc6dc986, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.965971 16244 raft_consensus.cc:348] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.966286 16244 raft_consensus.cc:374] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:12.966434 16244 raft_consensus.cc:729] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0ea263d9104e9d8869769234798319, State: Initialized, Role: FOLLOWER
I20260619 02:45:12.966399 16242 consensus_queue.cc:260] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.966763 16244 consensus_queue.cc:260] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:12.966961 16242 ts_tablet_manager.cc:1428] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:45:12.969125 16244 ts_tablet_manager.cc:1428] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 02:45:13.022480 16289 raft_consensus.cc:482] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:13.022709 16289 raft_consensus.cc:504] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:13.023345 16289 leader_election.cc:288] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:13.023995 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd5643da99c428abfaaa45918a378d8" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986" is_pre_election: true
I20260619 02:45:13.024072 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd5643da99c428abfaaa45918a378d8" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319" is_pre_election: true
I20260619 02:45:13.024246 15893 raft_consensus.cc:2454] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 0.
I20260619 02:45:13.024305 16172 raft_consensus.cc:2454] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 0.
I20260619 02:45:13.024737 15960 leader_election.cc:302] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, cb0ea263d9104e9d8869769234798319; no voters:
I20260619 02:45:13.025043 16289 raft_consensus.cc:2781] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:13.025153 16289 raft_consensus.cc:482] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:13.025265 16289 raft_consensus.cc:3037] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:13.027642 16289 raft_consensus.cc:504] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:13.028214 16289 leader_election.cc:288] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 election: Requested vote from peers 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641), cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363)
I20260619 02:45:13.028566 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd5643da99c428abfaaa45918a378d8" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8338ba0738f94e768c4d40a8bc6dc986"
I20260619 02:45:13.028746 15893 raft_consensus.cc:3037] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:13.028740 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd5643da99c428abfaaa45918a378d8" candidate_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0ea263d9104e9d8869769234798319"
I20260619 02:45:13.028961 16172 raft_consensus.cc:3037] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:13.030853 15893 raft_consensus.cc:2454] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 1.
I20260619 02:45:13.031018 16172 raft_consensus.cc:2454] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8ae65d3bbe4304832088cc3cc42e8f in term 1.
I20260619 02:45:13.031273 15960 leader_election.cc:302] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8ae65d3bbe4304832088cc3cc42e8f, 8338ba0738f94e768c4d40a8bc6dc986; no voters:
I20260619 02:45:13.031579 16289 raft_consensus.cc:2781] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:13.031966 16289 raft_consensus.cc:686] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 LEADER]: Becoming Leader. State: Replica: 7c8ae65d3bbe4304832088cc3cc42e8f, State: Running, Role: LEADER
I20260619 02:45:13.032533 16289 consensus_queue.cc:237] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } }
I20260619 02:45:13.035068 15730 catalog_manager.cc:5495] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66). New cstate: current_term: 1 leader_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8ae65d3bbe4304832088cc3cc42e8f" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 } health_report { overall_health: UNKNOWN } } }
I20260619 02:45:13.059257 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.054229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.7.246.66:44971)
0619 02:45:13.054747 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:13.054754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:13.054793 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:45:13.054853 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:13.054856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:13.054913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:13.055019 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:13.055434 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.056166 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.057503 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.057960 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.058105 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.058527 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:13.058627 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:13.058725 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:13.058879 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:45:13.058953 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":264,"threads_started":1}
I20260619 02:45:13.072201 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.067360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:42637 (local address 127.7.246.66:44971)
0619 02:45:13.067505 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:13.067509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:13.068045 (+ 536us) server_negotiation.cc:407] Connection header received
0619 02:45:13.068161 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:13.068165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:13.068220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:13.068290 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:13.068814 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.069528 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.070850 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.071239 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.071325 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.071653 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:13.071738 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:13.071799 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:13.071911 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:45:13.071972 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":45}
I20260619 02:45:13.072242 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.067299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44971 (local address 127.7.246.66:42637)
0619 02:45:13.067965 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:13.067980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:13.068085 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:13.068318 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:13.068321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:13.068333 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:45:13.068562 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:13.068569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.069685 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.069689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:13.070733 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:13.070745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.071385 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.071390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:13.071505 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.071538 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:45:13.071817 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:45:13.071825 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:45:13.071881 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:45:13.071973 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260619 02:45:13.080174 15893 raft_consensus.cc:1261] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Refusing update from remote peer 7c8ae65d3bbe4304832088cc3cc42e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:13.080173 16172 raft_consensus.cc:1261] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Refusing update from remote peer 7c8ae65d3bbe4304832088cc3cc42e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:13.080724 16345 consensus_queue.cc:1035] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0ea263d9104e9d8869769234798319" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:13.081066 16289 consensus_queue.cc:1035] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8338ba0738f94e768c4d40a8bc6dc986" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:13.130126 15730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:13.130359 15730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:13.133142 15730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a7a109df68a42e49c40f7c1a9595a98: Sending DeleteTablet for 3 replicas of tablet cdd5643da99c428abfaaa45918a378d8
I20260619 02:45:13.133996 15868 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd5643da99c428abfaaa45918a378d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:13 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:40520
I20260619 02:45:13.133993 16007 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd5643da99c428abfaaa45918a378d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:13 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38296
I20260619 02:45:13.134289 16319 tablet_replica.cc:330] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: stopping tablet replica
I20260619 02:45:13.134306 16318 tablet_replica.cc:330] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: stopping tablet replica
I20260619 02:45:13.134411 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd5643da99c428abfaaa45918a378d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:13 UTC) from {username='oryx', principal='oryx/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:53312
I20260619 02:45:13.134461 16319 raft_consensus.cc:2229] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:13.134498 16318 raft_consensus.cc:2229] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:13.134660 16319 raft_consensus.cc:2258] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:13.134972 16318 raft_consensus.cc:2258] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:13.135277 16317 tablet_replica.cc:330] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: stopping tablet replica
I20260619 02:45:13.135468 16317 raft_consensus.cc:2229] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:13.135672 16317 raft_consensus.cc:2258] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:13.135663 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15803
I20260619 02:45:13.136711 16319 ts_tablet_manager.cc:1905] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:13.136711 16317 ts_tablet_manager.cc:1905] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:13.140187 16318 ts_tablet_manager.cc:1905] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:13.143245 16317 ts_tablet_manager.cc:1918] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:13.143429 16317 log.cc:1196] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/cdd5643da99c428abfaaa45918a378d8
I20260619 02:45:13.143890 16317 ts_tablet_manager.cc:1939] T cdd5643da99c428abfaaa45918a378d8 P cb0ea263d9104e9d8869769234798319: Deleting consensus metadata
I20260619 02:45:13.145155 15717 catalog_manager.cc:4841] TS cb0ea263d9104e9d8869769234798319 (127.7.246.67:39363): tablet cdd5643da99c428abfaaa45918a378d8 (table test-table [id=939f64cd8369449c980deaae5d4f13a6]) successfully deleted
I20260619 02:45:13.146651 16318 ts_tablet_manager.cc:1918] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:13.146822 16318 log.cc:1196] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/cdd5643da99c428abfaaa45918a378d8
I20260619 02:45:13.147220 16318 ts_tablet_manager.cc:1939] T cdd5643da99c428abfaaa45918a378d8 P 7c8ae65d3bbe4304832088cc3cc42e8f: Deleting consensus metadata
I20260619 02:45:13.148162 15718 catalog_manager.cc:4841] TS 7c8ae65d3bbe4304832088cc3cc42e8f (127.7.246.66:44971): tablet cdd5643da99c428abfaaa45918a378d8 (table test-table [id=939f64cd8369449c980deaae5d4f13a6]) successfully deleted
I20260619 02:45:13.151091 16319 ts_tablet_manager.cc:1918] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:45:13.151368 16319 log.cc:1196] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/cdd5643da99c428abfaaa45918a378d8
I20260619 02:45:13.151891 16319 ts_tablet_manager.cc:1939] T cdd5643da99c428abfaaa45918a378d8 P 8338ba0738f94e768c4d40a8bc6dc986: Deleting consensus metadata
I20260619 02:45:13.152877 15717 catalog_manager.cc:4841] TS 8338ba0738f94e768c4d40a8bc6dc986 (127.7.246.65:33641): tablet cdd5643da99c428abfaaa45918a378d8 (table test-table [id=939f64cd8369449c980deaae5d4f13a6]) successfully deleted
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.65@KRBTEST.COM
I20260619 02:45:13.167689 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.139229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.7.246.65:33641)
0619 02:45:13.151723 (+ 12494us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:13.151732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:13.152264 (+ 532us) server_negotiation.cc:407] Connection header received
0619 02:45:13.152443 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:13.152450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:13.152539 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:13.152660 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:13.154050 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.155158 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.156339 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.156988 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.157119 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.160988 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:13.161008 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:13.161029 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:13.161069 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:13.164658 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.165519 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.165529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.165533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.165729 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.166120 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.166126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.166129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.166551 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:13.166859 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:13.167033 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 02:45:13.167242 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12361,"thread_start_us":12260,"threads_started":1}
I20260619 02:45:13.168315 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:13.320363 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15803
I20260619 02:45:13.343644 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15943
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.66@KRBTEST.COM
I20260619 02:45:13.359112 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.344854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.7.246.66:44971)
0619 02:45:13.345037 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:13.345042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:13.345056 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:45:13.345109 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:13.345113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:13.345168 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:13.345284 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:13.346518 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.347263 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.348595 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.349066 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.349214 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.353227 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:13.353241 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:13.353257 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:13.353297 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:13.356286 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.356951 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.356959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.356961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.357160 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.357660 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.357663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.357665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.358037 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:13.358270 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:13.358629 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 02:45:13.358792 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:45:13.359860 16073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:13.504122 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15943
I20260619 02:45:13.527040 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16082
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.7.246.67@KRBTEST.COM
I20260619 02:45:13.542063 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:13.527769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.7.246.67:39363)
0619 02:45:13.528234 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:13.528239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:13.528254 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:45:13.528303 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:13.528307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:13.528361 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:13.528667 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:13.530046 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.530717 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.531858 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:13.532210 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:13.532295 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:13.536569 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:13.536585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:13.536601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:13.536649 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:13.539350 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.540175 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.540183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.540185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.540336 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:13.540662 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:13.540666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:13.540668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:13.541032 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:13.541351 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:13.541594 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:45:13.541812 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260619 02:45:13.542718 16212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:13.668841 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16082
I20260619 02:45:13.689739 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 15700
I20260619 02:45:13.690811 15772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:13.803411 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 15700
2026-06-19T02:45:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6937 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[16366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[16366](info): set up 2 sockets
Jun 19 02:45:13 dist-test-slave-9tkz krb5kdc[16366](info): commencing operation
krb5kdc: starting...
W20260619 02:45:14.918758 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:45:15 dist-test-slave-9tkz krb5kdc[16366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:45:15.133400 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43701
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:37917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:15.446663 16379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:15.446935 16379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:15.447116 16379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:15.454218 16379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:15.454315 16379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:15.454439 16379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:15.454532 16379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:15.454615 16379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:15.464515 16379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:37917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:15.465890 16379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:15.467160 16379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:15.479224 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:15.479315 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:15.479247 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:15.480019 16379 server_base.cc:1034] running on GCE node
I20260619 02:45:15.480847 16379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:15.482412 16379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:15.483676 16379 hybrid_clock.cc:648] HybridClock initialized: now 1781837115483616 us; error 62 us; skew 500 ppm
I20260619 02:45:15.484108 16379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:45:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1746 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:45:15 dist-test-slave-9tkz krb5kdc[16393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:45:15 dist-test-slave-9tkz krb5kdc[16393](info): set up 2 sockets
Jun 19 02:45:15 dist-test-slave-9tkz krb5kdc[16393](info): commencing operation
krb5kdc: starting...
W20260619 02:45:16.654595 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:45:16 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:45:16.878870 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43749
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:17.200623 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:17.200912 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:17.201071 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:17.208249 16406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:17.208359 16406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:17.208441 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:17.208531 16406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:17.208607 16406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:17.218565 16406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:17.219861 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:17.221205 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:17.233088 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:17.233029 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:17.233174 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:17.234069 16406 server_base.cc:1034] running on GCE node
I20260619 02:45:17.234810 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:17.236181 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:17.237396 16406 hybrid_clock.cc:648] HybridClock initialized: now 1781837117237354 us; error 32 us; skew 500 ppm
I20260619 02:45:17.237795 16406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:17 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837117, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:17.244819 16406 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:45:17.246440 16406 webserver.cc:460] Webserver started at http://127.7.246.126:46167/ using document root <none> and password file <none>
I20260619 02:45:17.247165 16406 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:17.247311 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:17.247795 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:17.251022 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "143566975fe04fba895eec885bcd93a6"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "9b33c36df7d367d1ae8dbeb21fe8f889"
server_key_iv: "2bf985caa7acb341aae088581325194d"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.251853 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "143566975fe04fba895eec885bcd93a6"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "9b33c36df7d367d1ae8dbeb21fe8f889"
server_key_iv: "2bf985caa7acb341aae088581325194d"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.257009 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:45:17.260615 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:17.262105 16406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:17.262306 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "143566975fe04fba895eec885bcd93a6"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "9b33c36df7d367d1ae8dbeb21fe8f889"
server_key_iv: "2bf985caa7acb341aae088581325194d"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.262509 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:17.307106 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:17.311285 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:17.311602 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:17.330163 16406 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43749
I20260619 02:45:17.330183 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43749 every 8 connection(s)
I20260619 02:45:17.331589 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:17.335170 16473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:17.341367 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16406
I20260619 02:45:17.341624 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:45:17.342430 8153 external_mini_cluster.cc:1428] Setting key b119e947ddf94dfb84a7949835c2d2a3
I20260619 02:45:17.346208 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Bootstrap starting.
I20260619 02:45:17.349756 16473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:17.350858 16473 log.cc:824] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:17.354068 16473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: No bootstrap required, opened a new log
Jun 19 02:45:17 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:17.363133 16473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:17.363670 16473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:17.363870 16473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143566975fe04fba895eec885bcd93a6, State: Initialized, Role: FOLLOWER
I20260619 02:45:17.364610 16473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:17.364920 16473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:17.365074 16473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:17.365245 16473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:17.366288 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:17.344567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.7.246.126:43749)
0619 02:45:17.345329 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:17.345341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:17.345382 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:45:17.347190 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:17.347214 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:17.347609 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:17.347977 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:17.349887 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.351343 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.352915 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.353613 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.353755 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:17.358359 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:17.358367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:17.358382 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:17.358418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:17.361599 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:17.362347 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:17.362355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:17.362362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:17.362547 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:17.362937 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:17.362943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:17.362946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:17.363624 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:17.363926 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:17.364452 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 02:45:17.364916 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":356,"threads_started":1}
I20260619 02:45:17.367857 16473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:17.368381 16473 leader_election.cc:302] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 143566975fe04fba895eec885bcd93a6; no voters:
I20260619 02:45:17.369122 16473 leader_election.cc:288] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:45:17.369266 16478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:17.369987 16478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 LEADER]: Becoming Leader. State: Replica: 143566975fe04fba895eec885bcd93a6, State: Running, Role: LEADER
I20260619 02:45:17.370611 16478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:17.371316 16473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:17.373687 16479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "143566975fe04fba895eec885bcd93a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } } }
I20260619 02:45:17.373832 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 143566975fe04fba895eec885bcd93a6. Latest consensus state: current_term: 1 leader_uuid: "143566975fe04fba895eec885bcd93a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } } }
I20260619 02:45:17.374163 16479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:17.374257 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:17.376140 16484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:17.380818 16484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:17.388914 16484 catalog_manager.cc:1347] Generated new cluster ID: a85a5c67ab82427d8daf65040c465b26
I20260619 02:45:17.389060 16484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:17.408319 16484 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:45:17.409714 16484 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:17.425503 16484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Generated new TSK 0
I20260619 02:45:17.426442 16484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:45:17.482992 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:17.809412 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:17.809741 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:17.809909 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:17.817134 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:17.817270 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:17.817445 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:17.827224 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:17.828505 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:17.829797 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:17.842610 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:17.842655 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:17.842679 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:17.843858 16499 server_base.cc:1034] running on GCE node
I20260619 02:45:17.844398 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:17.845223 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:17.846411 16499 hybrid_clock.cc:648] HybridClock initialized: now 1781837117846364 us; error 33 us; skew 500 ppm
I20260619 02:45:17.846707 16499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:17 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837117, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:17.849915 16499 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:45:17.851354 16499 webserver.cc:460] Webserver started at http://127.7.246.65:41779/ using document root <none> and password file <none>
I20260619 02:45:17.852047 16499 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:17.852174 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:17.852530 16499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:17.855332 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "cefec18e83274977b4de37723a30a890"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "bedadb13ec1b45b521ce708e7fed579f"
server_key_iv: "1b12109481a13cae0c33a390e88f4b51"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.856048 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "cefec18e83274977b4de37723a30a890"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "bedadb13ec1b45b521ce708e7fed579f"
server_key_iv: "1b12109481a13cae0c33a390e88f4b51"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.860936 16499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:45:17.864331 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:17.865986 16499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:45:17.866153 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "cefec18e83274977b4de37723a30a890"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "bedadb13ec1b45b521ce708e7fed579f"
server_key_iv: "1b12109481a13cae0c33a390e88f4b51"
server_key_version: "encryptionkey@0"
I20260619 02:45:17.866348 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:17.903730 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:17.908113 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:17.908401 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:17.909349 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:17.910682 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:17.910799 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:17.910946 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:17.911017 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:17.944665 16499 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:38187
I20260619 02:45:17.944720 16626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:38187 every 8 connection(s)
I20260619 02:45:17.946022 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:17.954795 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16499
I20260619 02:45:17.955015 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:45:17.955600 8153 external_mini_cluster.cc:1428] Setting key 94f0f139c6316f9f0be45aa455c77db5
Jun 19 02:45:17 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837117, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:17.967603 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:17.948667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:41077 (local address 127.7.246.126:43749)
0619 02:45:17.949068 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:17.949073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:17.950107 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 02:45:17.952004 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:17.952009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:17.952100 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:17.952217 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:17.953713 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.954533 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.956364 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.956857 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.956957 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:17.960642 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:17.960651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:17.960658 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:17.960704 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:17.964265 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:17.965221 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:17.965224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:17.965227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:17.965367 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:17.965891 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:17.965895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:17.965897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:17.966152 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:17.966303 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:17.967141 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 02:45:17.967304 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":200,"threads_started":1}
I20260619 02:45:17.969069 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:17.949253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:41077)
0619 02:45:17.949951 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:17.949993 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:17.951743 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:17.952453 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:17.952462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:17.952952 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:17.953503 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:17.953520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.954704 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.954708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:17.956188 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:17.956203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:17.957083 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:17.957090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:17.957461 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:17.958281 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:17.958304 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:17.960429 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:17.964425 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:17.964437 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:17.964456 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:17.965101 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:17.965490 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:17.965496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:17.965499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:17.965772 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:17.966285 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:17.966296 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:17.966746 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 02:45:17.967041 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 02:45:17.967621 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":302,"threads_started":1}
I20260619 02:45:17.970661 16627 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43749
I20260619 02:45:17.971026 16627 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:17.971748 16627 heartbeater.cc:511] Master 127.7.246.126:43749 requested a full tablet report, sending...
I20260619 02:45:17.973863 16438 ts_manager.cc:194] Registered new tserver with Master: cefec18e83274977b4de37723a30a890 (127.7.246.65:38187)
I20260619 02:45:17.975342 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:41077
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:45:18.004851 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:18.319603 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:18.319912 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:18.320103 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:18.327030 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:18.327196 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:18.327397 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:18.336817 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:18.338104 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:18.339344 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:18.351950 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:18.352008 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:18.352035 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:18.354812 16633 server_base.cc:1034] running on GCE node
I20260619 02:45:18.355368 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:18.356194 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:18.357419 16633 hybrid_clock.cc:648] HybridClock initialized: now 1781837118357385 us; error 47 us; skew 500 ppm
I20260619 02:45:18.357764 16633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:18 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837118, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:18.360744 16633 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:45:18.362306 16633 webserver.cc:460] Webserver started at http://127.7.246.66:37697/ using document root <none> and password file <none>
I20260619 02:45:18.362964 16633 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:18.363098 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:18.363395 16633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:18.366232 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "3363d52747f246fe819bb4b7536cf1b5"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "386bc24fe541696fda316a9bf75484b9"
server_key_iv: "9038159c66fd3a15d06697245bed8a1f"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.366995 16633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "3363d52747f246fe819bb4b7536cf1b5"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "386bc24fe541696fda316a9bf75484b9"
server_key_iv: "9038159c66fd3a15d06697245bed8a1f"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.371731 16633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:45:18.374959 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.376205 16633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:45:18.376368 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "3363d52747f246fe819bb4b7536cf1b5"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "386bc24fe541696fda316a9bf75484b9"
server_key_iv: "9038159c66fd3a15d06697245bed8a1f"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.376554 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:18.404536 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:18.408730 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:18.409024 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:18.409925 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:18.411168 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:18.411263 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.411393 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:18.411478 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.443336 16633 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:45259
I20260619 02:45:18.443455 16760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:45259 every 8 connection(s)
I20260619 02:45:18.444603 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:18.444999 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16633
I20260619 02:45:18.445182 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:18.445775 8153 external_mini_cluster.cc:1428] Setting key 1241e865cf6b4345f01b40b1dd7eae93
Jun 19 02:45:18 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837118, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:18.465346 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:18.447166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:42199 (local address 127.7.246.126:43749)
0619 02:45:18.447347 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:18.447353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:18.448537 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 02:45:18.451439 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:18.451443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:18.451504 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:18.451586 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:18.453625 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.454311 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.456224 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.456636 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.456765 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:18.460004 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:18.460008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:18.460012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:18.460039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:18.462522 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:18.463180 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:18.463187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:18.463190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:18.463311 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:18.463742 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:18.463749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:18.463752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:18.463999 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:18.464158 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:18.464796 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 02:45:18.465001 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:45:18.466393 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:18.447616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:42199)
0619 02:45:18.448366 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:18.448422 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:18.451046 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:18.451832 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:18.451844 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:18.452622 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:18.453411 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:18.453437 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.454458 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.454466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:18.456082 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:18.456092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.456839 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.456846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:18.457063 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:18.457740 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:18.457761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:18.459787 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:18.462652 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:18.462660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:18.462675 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:18.463067 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:18.463455 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:18.463461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:18.463464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:18.463637 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:18.464120 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:18.464129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:18.464450 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 02:45:18.464730 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 02:45:18.465243 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":282,"threads_started":1}
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:45:18.467839 16761 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43749
I20260619 02:45:18.468113 16761 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:18.468766 16761 heartbeater.cc:511] Master 127.7.246.126:43749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:45:18.470207 16438 ts_manager.cc:194] Registered new tserver with Master: 3363d52747f246fe819bb4b7536cf1b5 (127.7.246.66:45259)
I20260619 02:45:18.471165 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:42199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:45:18.496299 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:18.826432 16766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:18.826731 16766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:18.826833 16766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:18.834239 16766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:18.834354 16766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:18.834483 16766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:18.845119 16766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:18.846467 16766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:18.847743 16766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:18.861447 16771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:18.861508 16772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:18.861531 16774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:18.862166 16766 server_base.cc:1034] running on GCE node
I20260619 02:45:18.862844 16766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:18.863762 16766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:18.864977 16766 hybrid_clock.cc:648] HybridClock initialized: now 1781837118864931 us; error 43 us; skew 500 ppm
I20260619 02:45:18.865290 16766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:18 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837118, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:18.868587 16766 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:45:18.870100 16766 webserver.cc:460] Webserver started at http://127.7.246.67:37007/ using document root <none> and password file <none>
I20260619 02:45:18.870832 16766 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:18.870975 16766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:18.871273 16766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:18.874454 16766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "8e72b90793e14c0299ddb247fac5e3dc"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "244a5ac94644fe699bc70a80a9f77331"
server_key_iv: "26de21852729c632b0d75ec7cd482858"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.875196 16766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "8e72b90793e14c0299ddb247fac5e3dc"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "244a5ac94644fe699bc70a80a9f77331"
server_key_iv: "26de21852729c632b0d75ec7cd482858"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.880445 16766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:45:18.883824 16781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.885059 16766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:18.885217 16766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "8e72b90793e14c0299ddb247fac5e3dc"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "244a5ac94644fe699bc70a80a9f77331"
server_key_iv: "26de21852729c632b0d75ec7cd482858"
server_key_version: "encryptionkey@0"
I20260619 02:45:18.885406 16766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:18.899708 16766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:18.903712 16766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:18.903957 16766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:18.904816 16766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:18.906138 16766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:18.906208 16766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.906288 16766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:18.906333 16766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:18.939010 16766 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:43365
I20260619 02:45:18.939044 16893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:43365 every 8 connection(s)
I20260619 02:45:18.940308 16766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:18.949869 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16766
I20260619 02:45:18.950060 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:18.950567 8153 external_mini_cluster.cc:1428] Setting key 0e6070e36c6ed443b1ed20aa83dd591b
Jun 19 02:45:18 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837118, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:18.959583 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:18.942851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:44003 (local address 127.7.246.126:43749)
0619 02:45:18.943020 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:18.943024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:18.944056 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 02:45:18.945903 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:18.945908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:18.945992 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:18.946109 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:18.947679 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.948508 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.950351 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.950826 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.950931 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:18.954590 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:18.954594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:18.954598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:18.954630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:18.956923 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:18.957604 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:18.957610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:18.957613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:18.957752 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:18.958183 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:18.958187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:18.958190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:18.958463 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:18.958578 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:18.959179 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 02:45:18.959334 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:45:18.960610 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:18.943214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:44003)
0619 02:45:18.943904 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:18.943946 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:18.945656 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:18.946360 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:18.946372 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:18.946845 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:18.947524 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:18.947540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.948656 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.948660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:18.950209 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:18.950225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:18.951045 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:18.951054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:18.951254 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:18.951939 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:18.951972 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:18.954418 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:18.957056 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:18.957062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:18.957074 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:18.957458 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:18.957842 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:18.957845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:18.957848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:18.958024 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:18.958568 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:18.958574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:18.958872 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 02:45:18.959078 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 02:45:18.959464 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":251,"threads_started":1}
I20260619 02:45:18.962015 16894 heartbeater.cc:347] Connected to a master server at 127.7.246.126:43749
I20260619 02:45:18.962330 16894 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:18.963009 16894 heartbeater.cc:511] Master 127.7.246.126:43749 requested a full tablet report, sending...
I20260619 02:45:18.964448 16438 ts_manager.cc:194] Registered new tserver with Master: 8e72b90793e14c0299ddb247fac5e3dc (127.7.246.67:43365)
I20260619 02:45:18.965425 16438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:44003
I20260619 02:45:18.966012 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:45:18.978431 16627 heartbeater.cc:503] Master 127.7.246.126:43749 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
Entry for principal oryx/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab.
WARNING: no policy specified for oryx/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
Entry for principal oryx/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab.
WARNING: no policy specified for oryx/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
Entry for principal oryx/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab.
I20260619 02:45:19.094322 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16499
Jun 19 02:45:19 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:45:19.109467 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:19.095354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.7.246.65:38187)
0619 02:45:19.095990 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:19.095997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:19.096024 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:45:19.096083 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:19.096089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:19.096323 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:19.096473 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:19.097792 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.098836 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.099987 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.100511 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.100602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:19.104404 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:19.104413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:19.104427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:19.104457 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:19.107148 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.107803 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.107810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.107812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.107922 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.108265 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.108268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.108270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.108568 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:19.108776 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:19.109033 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 02:45:19.109200 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":317,"threads_started":1}
I20260619 02:45:19.110783 16622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:19.193161 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16499
I20260619 02:45:19.209117 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16633
Jun 19 02:45:19 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:45:19.224959 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:19.210220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.7.246.66:45259)
0619 02:45:19.210717 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:19.210724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:19.210747 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:19.210801 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:19.210808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:19.211008 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:19.211125 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:19.212395 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.213393 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.214773 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.215393 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.215489 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:19.219804 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:19.219815 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:19.219828 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:19.219860 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:19.222667 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.223306 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.223312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.223314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.223431 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.223720 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.223723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.223724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.224014 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:19.224217 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:19.224487 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 02:45:19.224665 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":246,"threads_started":1}
I20260619 02:45:19.226308 16756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:19.325378 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16633
I20260619 02:45:19.345508 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16766
Jun 19 02:45:19 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:45:19.362241 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:19.346604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.7.246.67:43365)
0619 02:45:19.347327 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:19.347334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:19.347364 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:45:19.347421 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:19.347427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:19.347642 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:19.347838 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:19.349173 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.350356 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.351646 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:19.352172 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:19.352314 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:19.356839 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:19.356852 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:19.356866 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:19.356898 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:19.359695 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.360303 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.360310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.360312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.360433 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:19.360766 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:19.360770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:19.360772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:19.361129 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:19.361400 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:19.361631 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 02:45:19.361827 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":396,"threads_started":1}
I20260619 02:45:19.363564 16889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:19.444628 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16766
I20260619 02:45:19.460603 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16406
I20260619 02:45:19.461759 16468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:19.562938 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16406
I20260619 02:45:19.583252 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43749
--webserver_interface=127.7.246.126
--webserver_port=46167
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:19.905352 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:19.905735 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:19.905912 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:19.912914 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:19.913062 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:19.913169 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:19.913250 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:19.913313 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:19.923106 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=46167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:19.924463 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:19.925786 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:19.936944 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:19.936985 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:19.937026 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:19.938126 16911 server_base.cc:1034] running on GCE node
I20260619 02:45:19.938921 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:19.940276 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:19.941502 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781837119941465 us; error 52 us; skew 500 ppm
I20260619 02:45:19.941879 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:19 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837119, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:19.944975 16911 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:45:19.946689 16911 webserver.cc:460] Webserver started at http://127.7.246.126:46167/ using document root <none> and password file <none>
I20260619 02:45:19.947520 16911 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:19.947672 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:19.953096 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:45:19.956204 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:19.957604 16911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:45:19.957789 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "143566975fe04fba895eec885bcd93a6"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "9b33c36df7d367d1ae8dbeb21fe8f889"
server_key_iv: "2bf985caa7acb341aae088581325194d"
server_key_version: "encryptionkey@0"
I20260619 02:45:19.958760 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:19.984715 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:19.989265 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:19.989570 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:20.008682 16911 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43749
I20260619 02:45:20.008690 16977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43749 every 8 connection(s)
I20260619 02:45:20.010246 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:20.014214 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16911
I20260619 02:45:20.014952 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:38187
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:43749
--webserver_port=41779
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:45:20.017211 16978 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:45:20.020476 16978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Bootstrap starting.
I20260619 02:45:20.047566 16978 log.cc:824] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:20.058620 16978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:45:20.059230 16978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Bootstrap complete.
I20260619 02:45:20.073654 16978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:20.075044 16978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143566975fe04fba895eec885bcd93a6, State: Initialized, Role: FOLLOWER
I20260619 02:45:20.075825 16978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:20.076134 16978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:20.076265 16978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:20.076465 16978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:45:20.080786 16978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:20.081331 16978 leader_election.cc:302] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 143566975fe04fba895eec885bcd93a6; no voters:
I20260619 02:45:20.082034 16978 leader_election.cc:288] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:45:20.082203 16982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:45:20.083513 16982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [term 2 LEADER]: Becoming Leader. State: Replica: 143566975fe04fba895eec885bcd93a6, State: Running, Role: LEADER
I20260619 02:45:20.084148 16982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } }
I20260619 02:45:20.084403 16978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:20.089334 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "143566975fe04fba895eec885bcd93a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } } }
I20260619 02:45:20.089725 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:20.090147 16984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 143566975fe04fba895eec885bcd93a6. Latest consensus state: current_term: 2 leader_uuid: "143566975fe04fba895eec885bcd93a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143566975fe04fba895eec885bcd93a6" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43749 } } }
I20260619 02:45:20.090420 16984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:20.105683 16999 catalog_manager.cc:1259] Loaded cluster ID: a85a5c67ab82427d8daf65040c465b26
I20260619 02:45:20.105798 16999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: loading cluster ID for follower catalog manager: success
I20260619 02:45:20.109716 16999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: acquiring CA information for follower catalog manager: success
I20260619 02:45:20.112027 16999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:45:20.112300 16990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:20.114838 16990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:20.115981 16990 catalog_manager.cc:1259] Loaded cluster ID: a85a5c67ab82427d8daf65040c465b26
I20260619 02:45:20.116093 16990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:20.117847 16990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:20.119104 16990 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 143566975fe04fba895eec885bcd93a6: Loaded TSK: 0
I20260619 02:45:20.120107 16990 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:45:20.364398 16980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:20.364715 16980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:20.364892 16980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:20.372263 16980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:20.372417 16980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:20.372606 16980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:20.382577 16980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.65.keytab
--principal=oryx/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:38187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=41779
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:20.384008 16980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:20.385380 16980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:20.398689 17005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:20.398921 17006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:20.399962 17008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:20.400532 16980 server_base.cc:1034] running on GCE node
I20260619 02:45:20.401226 16980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:20.402302 16980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:20.403538 16980 hybrid_clock.cc:648] HybridClock initialized: now 1781837120403472 us; error 56 us; skew 500 ppm
I20260619 02:45:20.403864 16980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837120, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:20.407330 16980 init.cc:377] Logged in from keytab as oryx/127.7.246.65@KRBTEST.COM (short username oryx)
I20260619 02:45:20.409049 16980 webserver.cc:460] Webserver started at http://127.7.246.65:41779/ using document root <none> and password file <none>
I20260619 02:45:20.409823 16980 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:20.409974 16980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:20.415587 16980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:45:20.418440 17015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.419924 16980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:20.420089 16980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "cefec18e83274977b4de37723a30a890"
format_stamp: "Formatted at 2026-06-19 02:45:17 on dist-test-slave-9tkz"
server_key: "bedadb13ec1b45b521ce708e7fed579f"
server_key_iv: "1b12109481a13cae0c33a390e88f4b51"
server_key_version: "encryptionkey@0"
I20260619 02:45:20.420989 16980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:20.443800 16980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:20.448196 16980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:20.448472 16980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:20.449553 16980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:20.451047 16980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:20.451140 16980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.451288 16980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:20.451385 16980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.486229 16980 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:38187
I20260619 02:45:20.486433 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:38187 every 8 connection(s)
I20260619 02:45:20.487970 16980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:20.490670 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 16980
I20260619 02:45:20.491375 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:45259
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:43749
--webserver_port=37697
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:45:20.836272 17132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:20.836547 17132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:20.836691 17132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:20.844051 17132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:20.844169 17132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:20.844386 17132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:20.854091 17132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.66.keytab
--principal=oryx/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:45259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=37697
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:20.855466 17132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:20.856868 17132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:20.875613 17139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:20.876106 17138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:20.876256 17141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:20.877135 17132 server_base.cc:1034] running on GCE node
I20260619 02:45:20.877879 17132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:20.878945 17132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:20.880231 17132 hybrid_clock.cc:648] HybridClock initialized: now 1781837120880121 us; error 64 us; skew 500 ppm
I20260619 02:45:20.880609 17132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837120, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:20.884254 17132 init.cc:377] Logged in from keytab as oryx/127.7.246.66@KRBTEST.COM (short username oryx)
I20260619 02:45:20.886153 17132 webserver.cc:460] Webserver started at http://127.7.246.66:37697/ using document root <none> and password file <none>
I20260619 02:45:20.886898 17132 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:20.887054 17132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:20.893024 17132 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:45:20.897497 17148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.899438 17132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:45:20.899621 17132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "3363d52747f246fe819bb4b7536cf1b5"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "386bc24fe541696fda316a9bf75484b9"
server_key_iv: "9038159c66fd3a15d06697245bed8a1f"
server_key_version: "encryptionkey@0"
I20260619 02:45:20.900640 17132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:20.920993 17132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:20.925832 17132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:20.926126 17132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:20.927357 17132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:20.928877 17132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:20.928996 17132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.929140 17132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:20.929240 17132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:20.971312 17132 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:45259
I20260619 02:45:20.971364 17260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:45259 every 8 connection(s)
I20260619 02:45:20.972916 17132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:20.978171 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17132
I20260619 02:45:20.979148 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:43365
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:43749
--webserver_port=37007
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:20 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:45:21.329653 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:21.330009 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:21.330202 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:21.337888 17265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:21.338070 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:21.338280 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:21.348162 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/krb5kdc/oryx_127.7.246.67.keytab
--principal=oryx/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:43365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=37007
--tserver_master_addrs=127.7.246.126:43749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:21.349668 17265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:21.351024 17265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:21.367874 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:21.368610 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:21.372785 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:21.373723 17265 server_base.cc:1034] running on GCE node
I20260619 02:45:21.374433 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:21.375612 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:21.376863 17265 hybrid_clock.cc:648] HybridClock initialized: now 1781837121376798 us; error 63 us; skew 500 ppm
I20260619 02:45:21.377194 17265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:21 dist-test-slave-9tkz krb5kdc[16393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837121, etypes {rep=17 tkt=17 ses=17}, oryx/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:21.380530 17265 init.cc:377] Logged in from keytab as oryx/127.7.246.67@KRBTEST.COM (short username oryx)
I20260619 02:45:21.382476 17265 webserver.cc:460] Webserver started at http://127.7.246.67:37007/ using document root <none> and password file <none>
I20260619 02:45:21.383247 17265 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:21.383404 17265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:21.389050 17265 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:45:21.391815 17281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:21.393287 17265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:21.393461 17265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "8e72b90793e14c0299ddb247fac5e3dc"
format_stamp: "Formatted at 2026-06-19 02:45:18 on dist-test-slave-9tkz"
server_key: "244a5ac94644fe699bc70a80a9f77331"
server_key_iv: "26de21852729c632b0d75ec7cd482858"
server_key_version: "encryptionkey@0"
I20260619 02:45:21.394466 17265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:21.426051 17265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:21.430366 17265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:21.430650 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:21.431695 17265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:21.433135 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:21.433260 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:21.433393 17265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:21.433527 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:21.474622 17265 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:43365
I20260619 02:45:21.474668 17393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:43365 every 8 connection(s)
I20260619 02:45:21.476259 17265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:21.486680 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17265
Jun 19 02:45:21 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:21 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:21.527134 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:21.488194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.7.246.126:43749)
0619 02:45:21.504420 (+ 16226us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:21.504424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:21.504440 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:45:21.504526 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:21.504533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:21.504610 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:21.504739 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:21.510101 (+ 5362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.511394 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.512719 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.513388 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.513511 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:21.516886 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:21.516898 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:21.516920 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:21.516976 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:21.521353 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:21.522343 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:21.522352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:21.522363 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:21.522560 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:21.523065 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:21.523070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:21.523074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:21.523760 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:21.525161 (+ 1401us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:21.525455 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:45:21.525974 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16105,"thread_start_us":13334,"threads_started":1}
W20260619 02:45:22.033576 17131 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.529s user 0.008s sys 0.007s
W20260619 02:45:22.035699 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:20.491247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:54725 (local address 127.7.246.126:43749)
0619 02:45:20.492260 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:20.492284 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:20.492803 (+ 519us) server_negotiation.cc:407] Connection header received
0619 02:45:20.495774 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:20.495796 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:20.496261 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:20.496705 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:20.498479 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.500047 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.502744 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.503404 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.503540 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.035264 (+1531724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:54725: BlockingRecv error: recv got EOF from 127.7.246.65:54725 (error 108)
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":545,"threads_started":1}
I20260619 02:45:22.036578 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:20.491532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:54725)
0619 02:45:20.492610 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:20.492647 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:20.495109 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:20.496847 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:20.496858 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:20.497584 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:20.498277 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:20.498295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.500219 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.500226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:20.502561 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:20.502579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.503642 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.503650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:20.503979 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:20.504713 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:20.504755 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.034907 (+1530152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":613,"threads_started":1}
W20260619 02:45:22.038038 17128 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:43749 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.246.126:43749: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:45:22.055917 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.039190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:44713 (local address 127.7.246.126:43749)
0619 02:45:22.039427 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.039437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.039574 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:45:22.039835 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.039843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.039961 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.040115 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.041416 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.043152 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.046024 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.047015 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.047199 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.055523 (+ 8324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:44713: BlockingRecv error: recv got EOF from 127.7.246.65:44713 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:45:22.058369 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.039079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:44713)
0619 02:45:22.039445 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.039472 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.039648 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.040174 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.040183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.040906 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.041217 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.041236 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.043409 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.043418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.045812 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.045835 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.047260 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.047266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.047531 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.048541 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.048578 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.055207 (+ 6629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193}
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:22.079224 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.059309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:34573)
0619 02:45:22.061869 (+ 2560us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.061906 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.062186 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.063006 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.063014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.063915 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.064230 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.064247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.066442 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.066450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.069371 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.069394 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.071085 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.071094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.071379 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.072370 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.072417 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.078761 (+ 6344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2403,"thread_start_us":12804,"threads_started":1}
W20260619 02:45:22.079473 17130 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.059439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:34573 (local address 127.7.246.126:43749)
0619 02:45:22.059668 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.059677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.062002 (+ 2325us) server_negotiation.cc:407] Connection header received
0619 02:45:22.062415 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.062423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.062598 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.062695 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.064459 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.066159 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.069626 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.070574 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.070721 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.079103 (+ 8382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:34573: BlockingRecv error: recv got EOF from 127.7.246.65:34573 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 02:45:22.079841 17128 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:45:22.426847 17264 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.441s user 0.000s sys 0.017s
W20260619 02:45:22.428404 17263 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:20.975911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:37459 (local address 127.7.246.126:43749)
0619 02:45:20.976567 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:20.976577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:20.977596 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 02:45:20.979795 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:20.979802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:20.979901 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:20.980028 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:20.981580 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.982415 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.984038 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.984546 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.984697 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.428168 (+1443471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:37459: BlockingRecv error: recv got EOF from 127.7.246.66:37459 (error 108)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":298,"threads_started":1}
I20260619 02:45:22.429782 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:20.976474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:37459)
0619 02:45:20.977326 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:20.977378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:20.979480 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:20.980291 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:20.980300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:20.980816 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:20.981370 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:20.981387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.982584 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.982589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:20.983876 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:20.983888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:20.984795 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:20.984805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:20.985078 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:20.985870 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:20.985894 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.428043 (+1442149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":368,"threads_started":1}
W20260619 02:45:22.430732 17261 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:43749 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.246.126:43749: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:22.445725 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.431324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:51459)
0619 02:45:22.431595 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.431623 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.431810 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.432157 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.432165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.432813 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.433098 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.433114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.434284 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.434293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.436579 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.436599 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.437339 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.437347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.437630 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.441433 (+ 3803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.441452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.445371 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 02:45:22.445837 17263 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.431451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:51459 (local address 127.7.246.126:43749)
0619 02:45:22.431583 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.431587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.431739 (+ 152us) server_negotiation.cc:407] Connection header received
0619 02:45:22.431943 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.431946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.432006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.432086 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.433306 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.434119 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.436766 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.437139 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.437227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.445609 (+ 8382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:51459: BlockingRecv error: recv got EOF from 127.7.246.66:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:22.456859 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.446717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:58381)
0619 02:45:22.446871 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.446894 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.447003 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.447669 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.447672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.447942 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.448079 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.448086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.450003 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.450009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.451360 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.451372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.452187 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.452193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.452381 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.452967 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.452984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.456579 (+ 3595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 02:45:22.457029 17263 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.447022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:58381 (local address 127.7.246.126:43749)
0619 02:45:22.447260 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.447269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.447292 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:22.447364 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.447372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.447523 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.447664 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.448210 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.449829 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.451483 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.451990 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.452164 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.456758 (+ 4594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:58381: BlockingRecv error: recv got EOF from 127.7.246.66:58381 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 02:45:22.457523 17261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:45:22.671343 17396 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.152s user 0.003s sys 0.012s
W20260619 02:45:22.672533 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:21.481044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:56643 (local address 127.7.246.126:43749)
0619 02:45:21.501826 (+ 20782us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:21.501836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:21.508763 (+ 6927us) server_negotiation.cc:407] Connection header received
0619 02:45:21.508813 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:21.508817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:21.508874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:21.508998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:21.511184 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.514270 (+ 3086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.516898 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.517563 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.517739 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.672275 (+1154536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:56643: BlockingRecv error: recv got EOF from 127.7.246.67:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":20620,"thread_start_us":23199,"threads_started":1}
I20260619 02:45:22.673144 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:21.480385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:56643)
0619 02:45:21.506530 (+ 26145us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:21.506610 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:21.508417 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:21.509297 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:21.509310 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:21.510110 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:21.510954 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:21.510981 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.514463 (+ 3482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.514468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:21.516750 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:21.516764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:21.518104 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:21.518112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:21.518435 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:21.519173 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:21.519205 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.672119 (+1152914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25732,"thread_start_us":28666,"threads_started":1}
W20260619 02:45:22.673987 17394 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:43749 (0 consecutive failures): Not authorized: Failed to ping master at 127.7.246.126:43749: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:22.683984 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.674626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:39627)
0619 02:45:22.674839 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.674861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.674985 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.675588 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.675593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.675971 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.676125 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.676133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.677195 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.677200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.678233 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.678250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.678954 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.678960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.679118 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.679825 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.679842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.683672 (+ 3830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:45:22.684111 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.674708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:39627 (local address 127.7.246.126:43749)
0619 02:45:22.674884 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.674892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.674912 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:45:22.675372 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.675378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.675453 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.675560 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.676290 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.677049 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.678379 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.678782 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.678924 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.683879 (+ 4955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:39627: BlockingRecv error: recv got EOF from 127.7.246.67:39627 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:22 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:22.694360 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:22.684910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:50247)
0619 02:45:22.685059 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:22.685075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:22.685157 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:22.685632 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:22.685637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:22.685957 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:22.686133 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.686156 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.687122 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.687127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.688096 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:22.688106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.689100 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.689108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:22.689376 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.690182 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:22.690205 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:22.694076 (+ 3871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 02:45:22.694540 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:22.685125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:50247 (local address 127.7.246.126:43749)
0619 02:45:22.685356 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:22.685361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:22.685374 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:45:22.685411 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:22.685415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:22.685479 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:22.685591 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:22.686272 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.686991 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.688228 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:22.688890 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:22.689017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:22.694339 (+ 5322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:50247: BlockingRecv error: recv got EOF from 127.7.246.67:50247 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260619 02:45:22.695108 17394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:23.088868 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:23.080362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:33725)
0619 02:45:23.080845 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:23.080864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:23.080973 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:23.081271 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:23.081274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:23.081717 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:23.081863 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.081872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.082812 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.082816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.083758 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.083769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.084446 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.084451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.084569 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.085130 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:23.085147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:23.088516 (+ 3369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":280,"threads_started":1}
W20260619 02:45:23.088922 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:23.080398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:33725 (local address 127.7.246.126:43749)
0619 02:45:23.080559 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:23.080564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:23.080933 (+ 369us) server_negotiation.cc:407] Connection header received
0619 02:45:23.081069 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:23.081073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:23.081146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:23.081237 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:23.081988 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.082690 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.083878 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.084295 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.084385 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.088710 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:33725: BlockingRecv error: recv got EOF from 127.7.246.65:33725 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:23.466852 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:23.458097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:45341)
0619 02:45:23.458649 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:23.458674 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:23.458786 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:23.459076 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:23.459079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:23.459470 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:23.459605 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.459612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.460612 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.460616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.461534 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.461542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.462446 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.462450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.462561 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.463130 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:23.463146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:23.466484 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":338,"threads_started":1}
W20260619 02:45:23.466925 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:23.458136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:45341 (local address 127.7.246.126:43749)
0619 02:45:23.458331 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:23.458335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:23.458784 (+ 449us) server_negotiation.cc:407] Connection header received
0619 02:45:23.458886 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:23.458890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:23.458952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:23.459056 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:23.459731 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.460486 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.461695 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.462229 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.462386 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.466666 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:45341: BlockingRecv error: recv got EOF from 127.7.246.66:45341 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:23 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:23.705384 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:23.695643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:53807)
0619 02:45:23.696166 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:23.696185 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:23.696355 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:23.696674 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:23.696679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:23.697076 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:23.697210 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.697218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.698655 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.698667 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.699759 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:23.699772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.700596 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.700605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:23.700788 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.701387 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:23.701404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:23.705067 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":309,"threads_started":1}
W20260619 02:45:23.705665 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:23.695717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:53807 (local address 127.7.246.126:43749)
0619 02:45:23.695950 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:23.695958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:23.696277 (+ 319us) server_negotiation.cc:407] Connection header received
0619 02:45:23.696470 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:23.696477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:23.696550 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:23.696646 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:23.697343 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.698495 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.699906 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:23.700382 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:23.700534 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:23.705377 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:53807: BlockingRecv error: recv got EOF from 127.7.246.67:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:24.098794 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:24.089999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:59789)
0619 02:45:24.090485 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:24.090501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:24.090676 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:24.091016 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:24.091021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:24.091442 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:24.091600 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.091608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.092593 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.092600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.093513 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.093523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.094215 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.094221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.094394 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.094926 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:24.094942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:24.098527 (+ 3585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
W20260619 02:45:24.098929 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:24.090034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:59789 (local address 127.7.246.126:43749)
0619 02:45:24.090230 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:24.090234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:24.090592 (+ 358us) server_negotiation.cc:407] Connection header received
0619 02:45:24.090802 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:24.090807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:24.090887 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:24.090983 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:24.091739 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.092463 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.093672 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.094068 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.094197 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.098702 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:59789: BlockingRecv error: recv got EOF from 127.7.246.65:59789 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:24.476817 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:24.467850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:41303)
0619 02:45:24.468378 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:24.468394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:24.468502 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:24.468790 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:24.468794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:24.469186 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:24.469374 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.469384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.470357 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.470361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.471278 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.471288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.471933 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.471937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.472058 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.472725 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:24.472743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:24.476532 (+ 3789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":301,"threads_started":1}
W20260619 02:45:24.476980 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:24.467912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:41303 (local address 127.7.246.126:43749)
0619 02:45:24.468101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:24.468106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:24.468466 (+ 360us) server_negotiation.cc:407] Connection header received
0619 02:45:24.468597 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:24.468600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:24.468662 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:24.468741 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:24.469499 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.470237 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.471399 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.471805 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.471896 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.476760 (+ 4864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:41303: BlockingRecv error: recv got EOF from 127.7.246.66:41303 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:24 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:24.715651 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:24.706348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:60671)
0619 02:45:24.706864 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:24.706884 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:24.707035 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:24.707383 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:24.707391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:24.707808 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:24.707995 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.708004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.709046 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.709054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.710336 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:24.710358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.711150 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.711156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:24.711323 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.711874 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:24.711889 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:24.715304 (+ 3415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
W20260619 02:45:24.715730 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:24.706542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:60671 (local address 127.7.246.126:43749)
0619 02:45:24.706759 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:24.706765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:24.706964 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:45:24.707154 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:24.707160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:24.707243 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:24.707344 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:24.708137 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.708911 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.710478 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:24.711003 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:24.711145 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:24.715500 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:60671: BlockingRecv error: recv got EOF from 127.7.246.67:60671 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:25.110956 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:25.099659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:34329)
0619 02:45:25.100115 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:25.100129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:25.100292 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:25.100628 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:25.100632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:25.101150 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:25.101374 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.101389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.102841 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.102847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.104533 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.104562 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.105430 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.105436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.105652 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.106438 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:25.106459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:25.110590 (+ 4131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":226,"threads_started":1}
W20260619 02:45:25.111119 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:25.099728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:34329 (local address 127.7.246.126:43749)
0619 02:45:25.099924 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:25.099933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:25.100205 (+ 272us) server_negotiation.cc:407] Connection header received
0619 02:45:25.100427 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:25.100438 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:25.100507 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:25.100597 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:25.101603 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.102658 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.104719 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.105281 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.105417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.110889 (+ 5472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:34329: BlockingRecv error: recv got EOF from 127.7.246.65:34329 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:25.486673 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:25.478048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:50619)
0619 02:45:25.478612 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:25.478632 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:25.478768 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:25.479098 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:25.479103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:25.479478 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:25.479651 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.479659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.480643 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.480649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.481570 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.481580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.482247 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.482252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.482433 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.483043 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:25.483059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:25.486460 (+ 3401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":325,"threads_started":1}
W20260619 02:45:25.486953 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:25.478092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:50619 (local address 127.7.246.126:43749)
0619 02:45:25.478273 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:25.478280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:25.478722 (+ 442us) server_negotiation.cc:407] Connection header received
0619 02:45:25.478912 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:25.478917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:25.478977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:25.479072 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:25.479790 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.480502 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.481708 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.482116 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.482239 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.486703 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:50619: BlockingRecv error: recv got EOF from 127.7.246.66:50619 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:25 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:25.726028 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:25.716554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:36071)
0619 02:45:25.717091 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:25.717111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:25.717299 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:25.717677 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:25.717682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:25.718079 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:25.718258 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.718267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.719300 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.719307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.720220 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:25.720231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.720937 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.720942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:25.721140 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.721865 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:25.721881 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:25.725661 (+ 3780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
W20260619 02:45:25.726122 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:25.716701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:36071 (local address 127.7.246.126:43749)
0619 02:45:25.716920 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:25.716927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:25.717197 (+ 270us) server_negotiation.cc:407] Connection header received
0619 02:45:25.717426 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:25.717432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:25.717508 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:25.717650 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:25.718412 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.719153 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.720363 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:25.720792 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:25.720925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:25.725870 (+ 4945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:36071: BlockingRecv error: recv got EOF from 127.7.246.67:36071 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:26.120158 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:26.111854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:33489)
0619 02:45:26.112416 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:26.112432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:26.112547 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:26.112816 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:26.112820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:26.113178 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:26.113330 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.113338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.114336 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.114340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.115298 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.115308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.115914 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.115918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.116029 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.116587 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:26.116604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:26.119912 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":340,"threads_started":1}
W20260619 02:45:26.120393 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:26.111897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:33489 (local address 127.7.246.126:43749)
0619 02:45:26.112070 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:26.112076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:26.112506 (+ 430us) server_negotiation.cc:407] Connection header received
0619 02:45:26.112634 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:26.112638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:26.112699 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:26.112779 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:26.113446 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.114215 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.115415 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.115800 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.115888 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.120105 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:33489: BlockingRecv error: recv got EOF from 127.7.246.65:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:26.496575 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:26.487794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:43391)
0619 02:45:26.488410 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:26.488426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:26.488587 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:26.488916 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:26.488920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:26.489293 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:26.489445 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.489455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.490494 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.490501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.491483 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.491494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.492211 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.492217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.492388 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.492916 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:26.492932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:26.496215 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":416,"threads_started":1}
W20260619 02:45:26.496614 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:26.487892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:43391 (local address 127.7.246.126:43749)
0619 02:45:26.488063 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:26.488071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:26.488512 (+ 441us) server_negotiation.cc:407] Connection header received
0619 02:45:26.488710 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:26.488716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:26.488791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:26.488888 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:26.489610 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.490364 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.491625 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.492051 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.492196 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.496414 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:43391: BlockingRecv error: recv got EOF from 127.7.246.66:43391 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:26 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:26.735491 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:26.726874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:42923)
0619 02:45:26.727349 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:26.727368 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:26.727517 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:26.727854 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:26.727858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:26.728252 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:26.728411 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.728419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.729434 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.729440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.730462 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:26.730473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.731153 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.731158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:26.731341 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.731982 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:26.731998 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:26.735131 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
W20260619 02:45:26.735598 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:26.726932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:42923 (local address 127.7.246.126:43749)
0619 02:45:26.727152 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:26.727160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:26.727453 (+ 293us) server_negotiation.cc:407] Connection header received
0619 02:45:26.727656 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:26.727660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:26.727723 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:26.727822 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:26.728555 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.729282 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.730604 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:26.731007 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:26.731143 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:26.735355 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:42923: BlockingRecv error: recv got EOF from 127.7.246.67:42923 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:27.129817 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:27.121097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:55451)
0619 02:45:27.121568 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:27.121583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:27.121692 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:27.121981 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:27.121984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:27.122383 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:27.122522 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.122530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.123693 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.123698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.124806 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.124819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.125448 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.125452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.125629 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.126327 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:27.126344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:27.129596 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":268,"threads_started":1}
W20260619 02:45:27.130053 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:27.121152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:55451 (local address 127.7.246.126:43749)
0619 02:45:27.121379 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:27.121387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:27.121661 (+ 274us) server_negotiation.cc:407] Connection header received
0619 02:45:27.121790 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:27.121793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:27.121853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:27.121934 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:27.122689 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.123547 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.124938 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.125311 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.125427 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.129804 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:55451: BlockingRecv error: recv got EOF from 127.7.246.65:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:27.509745 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:27.497492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:50871)
0619 02:45:27.498025 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:27.498039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:27.498207 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:27.498746 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:27.498750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:27.499374 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:27.499565 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.499577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.500608 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.500611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.501476 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.501484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.502132 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.502136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.502250 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.506216 (+ 3966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:27.506233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:27.509476 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":312,"threads_started":1}
W20260619 02:45:27.509948 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:27.497720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:50871 (local address 127.7.246.126:43749)
0619 02:45:27.497929 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:27.497936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:27.498132 (+ 196us) server_negotiation.cc:407] Connection header received
0619 02:45:27.498334 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:27.498339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:27.498434 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:27.498534 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:27.499730 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.500475 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.501619 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.502021 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.502112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.509716 (+ 7604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:50871: BlockingRecv error: recv got EOF from 127.7.246.66:50871 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:27 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:27.744865 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:27.736536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:36269)
0619 02:45:27.737043 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:27.737063 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:27.737169 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:27.737444 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:27.737447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:27.737850 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:27.738004 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.738012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.739142 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.739146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.740038 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:27.740048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.740688 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.740692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:27.740811 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.741378 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:27.741394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:27.744618 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":280,"threads_started":1}
W20260619 02:45:27.745025 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:27.736620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:36269 (local address 127.7.246.126:43749)
0619 02:45:27.736814 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:27.736819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:27.737203 (+ 384us) server_negotiation.cc:407] Connection header received
0619 02:45:27.737258 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:27.737261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:27.737323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:27.737403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:27.738135 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.739026 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.740173 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:27.740574 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:27.740665 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:27.744814 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:36269: BlockingRecv error: recv got EOF from 127.7.246.67:36269 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:28.139132 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:28.130736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:47511)
0619 02:45:28.131248 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:28.131270 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:28.131383 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:28.131674 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:28.131677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:28.132030 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:28.132184 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.132192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.133103 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.133107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.134091 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.134102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.134814 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.134820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.134994 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.135594 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:28.135611 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:28.138807 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
W20260619 02:45:28.139240 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:28.130773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:47511 (local address 127.7.246.126:43749)
0619 02:45:28.130986 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:28.130991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:28.131349 (+ 358us) server_negotiation.cc:407] Connection header received
0619 02:45:28.131480 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:28.131483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:28.131541 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:28.131615 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:28.132297 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.132985 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.134246 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.134663 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.134801 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.139003 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:47511: BlockingRecv error: recv got EOF from 127.7.246.65:47511 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:28.519496 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:28.510828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:44797)
0619 02:45:28.511343 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:28.511358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:28.511520 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:28.511913 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:28.511918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:28.512439 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:28.512671 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.512684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.513759 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.513763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.514650 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.514660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.515339 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.515343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.515461 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.516000 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:28.516016 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:28.519278 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
W20260619 02:45:28.519655 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:28.510906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44797 (local address 127.7.246.126:43749)
0619 02:45:28.511210 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:28.511216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:28.511452 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:45:28.511657 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:28.511662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:28.511753 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:28.511842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:28.512836 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.513652 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.514769 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.515199 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.515302 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.519459 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:44797: BlockingRecv error: recv got EOF from 127.7.246.66:44797 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:28 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:28.754354 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:28.745727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:58087)
0619 02:45:28.746160 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:28.746182 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:28.746354 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:28.746680 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:28.746686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:28.747108 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:28.747270 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.747278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.748313 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.748319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.749233 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:28.749244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.749994 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.750000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:28.750179 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.750848 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:28.750865 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:28.754077 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":252,"threads_started":1}
W20260619 02:45:28.754523 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:28.745834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:58087 (local address 127.7.246.126:43749)
0619 02:45:28.746071 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:28.746077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:28.746286 (+ 209us) server_negotiation.cc:407] Connection header received
0619 02:45:28.746474 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:28.746479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:28.746548 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:28.746643 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:28.747411 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.748149 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.749377 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:28.749848 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:28.749978 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:28.754270 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:58087: BlockingRecv error: recv got EOF from 127.7.246.67:58087 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:45:29.150530 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:29.140216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:44083 (local address 127.7.246.126:43749)
0619 02:45:29.140421 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:29.140426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:29.140716 (+ 290us) server_negotiation.cc:407] Connection header received
0619 02:45:29.140893 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:29.140897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:29.140965 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:29.141048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:29.141887 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.142633 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.143782 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.144193 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.144284 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.150243 (+ 5959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:44083: BlockingRecv error: recv got EOF from 127.7.246.65:44083 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:45:29.151203 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:29.140052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:44083)
0619 02:45:29.140625 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:29.140641 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:29.140749 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:29.141091 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:29.141094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:29.141497 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:29.141747 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.141756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.142745 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.142748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.143669 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.143679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.144321 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.144325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.144432 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.145097 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:29.145114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:29.150783 (+ 5669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":275,"threads_started":1}
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:29.529827 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:29.520599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:41613)
0619 02:45:29.521252 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:29.521269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:29.521401 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:29.521723 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:29.521725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:29.522166 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:29.522358 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.522366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.523334 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.523338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.524457 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.524468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.525166 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.525170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.525350 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.525963 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:29.525979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:29.529439 (+ 3460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":288,"threads_started":1}
W20260619 02:45:29.530010 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:29.520655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:41613 (local address 127.7.246.126:43749)
0619 02:45:29.520821 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:29.520826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:29.521374 (+ 548us) server_negotiation.cc:407] Connection header received
0619 02:45:29.521510 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:29.521513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:29.521595 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:29.521680 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:29.522482 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.523206 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.524580 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.524999 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.525093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.529748 (+ 4655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:41613: BlockingRecv error: recv got EOF from 127.7.246.66:41613 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:29 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:29.763942 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:29.755234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:40177)
0619 02:45:29.755888 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:29.755905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:29.756011 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:29.756299 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:29.756302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:29.756660 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:29.756824 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.756832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.757770 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.757773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.758692 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:29.758702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.759356 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.759360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:29.759536 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.760083 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:29.760099 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:29.763618 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":284,"threads_started":1}
W20260619 02:45:29.764032 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:29.755354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:40177 (local address 127.7.246.126:43749)
0619 02:45:29.755537 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:29.755543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:29.755984 (+ 441us) server_negotiation.cc:407] Connection header received
0619 02:45:29.756105 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:29.756108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:29.756174 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:29.756256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:29.756947 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.757667 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.758812 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:29.759231 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:29.759324 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:29.763819 (+ 4495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:40177: BlockingRecv error: recv got EOF from 127.7.246.67:40177 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:30.161491 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:30.152566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:41655)
0619 02:45:30.153135 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:30.153157 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:30.153296 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:30.153768 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:30.153773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:30.154309 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:30.154471 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.154480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.155501 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.155506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.156450 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.156479 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.157210 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.157215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.157376 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.158047 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:30.158065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:30.161252 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":226,"threads_started":1}
W20260619 02:45:30.161712 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:30.152691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:41655 (local address 127.7.246.126:43749)
0619 02:45:30.152922 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:30.152929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:30.153244 (+ 315us) server_negotiation.cc:407] Connection header received
0619 02:45:30.153457 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:30.153464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:30.153554 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:30.153737 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:30.154608 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.155350 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.156629 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.157034 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.157168 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.161430 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:41655: BlockingRecv error: recv got EOF from 127.7.246.65:41655 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:30.539753 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:30.530908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:43985)
0619 02:45:30.531425 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:30.531446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:30.531592 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:30.531968 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:30.531972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:30.532391 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:30.532546 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.532554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.533823 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.533829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.534745 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.534755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.535510 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.535515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.535684 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.536315 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:30.536332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:30.539521 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":365,"threads_started":1}
W20260619 02:45:30.539942 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:30.531015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:43985 (local address 127.7.246.126:43749)
0619 02:45:30.531277 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:30.531284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:30.531542 (+ 258us) server_negotiation.cc:407] Connection header received
0619 02:45:30.531753 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:30.531758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:30.531833 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:30.531934 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:30.532692 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.533683 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.534888 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.535337 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.535481 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.539725 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:43985: BlockingRecv error: recv got EOF from 127.7.246.66:43985 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:30 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:30.774569 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:30.764787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:34431)
0619 02:45:30.765372 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:30.765395 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:30.765609 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:30.766002 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:30.766008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:30.766482 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:30.766675 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.766686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.767861 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.767870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.768947 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:30.768960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.769892 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.769900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:30.770094 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.770715 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:30.770737 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:30.774218 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":297,"threads_started":1}
W20260619 02:45:30.774636 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:30.764899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:34431 (local address 127.7.246.126:43749)
0619 02:45:30.765160 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:30.765168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:30.765494 (+ 326us) server_negotiation.cc:407] Connection header received
0619 02:45:30.765756 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:30.765770 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:30.765842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:30.765970 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:30.766835 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.767705 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.769123 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:30.769687 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:30.769852 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:30.774407 (+ 4555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:34431: BlockingRecv error: recv got EOF from 127.7.246.67:34431 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:31.171043 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:31.162406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:58353)
0619 02:45:31.162911 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:31.162926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:31.163078 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:31.163436 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:31.163440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:31.163834 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:31.163966 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.163973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.164912 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.164916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.165900 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.165910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.166577 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.166581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.166691 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.167235 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:31.167251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:31.170723 (+ 3472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":264,"threads_started":1}
W20260619 02:45:31.171218 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:31.162471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:58353 (local address 127.7.246.126:43749)
0619 02:45:31.162647 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:31.162652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:31.163003 (+ 351us) server_negotiation.cc:407] Connection header received
0619 02:45:31.163209 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:31.163213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:31.163273 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:31.163350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:31.164077 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.164801 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.166040 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.166456 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.166549 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.170922 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:58353: BlockingRecv error: recv got EOF from 127.7.246.65:58353 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:31.549507 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:31.540681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:48757)
0619 02:45:31.541143 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:31.541159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:31.541260 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:31.541534 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:31.541537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:31.541955 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:31.542089 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.542097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.543010 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.543013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.544019 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.544030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.544724 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.544728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.544854 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.545430 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:31.545446 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:31.549138 (+ 3692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":211,"threads_started":1}
W20260619 02:45:31.549530 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:31.540688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:48757 (local address 127.7.246.126:43749)
0619 02:45:31.540914 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:31.540924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:31.541235 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:45:31.541350 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:31.541353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:31.541411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:31.541486 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:31.542202 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.542906 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.544166 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.544591 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.544705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.549305 (+ 4600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:48757: BlockingRecv error: recv got EOF from 127.7.246.66:48757 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:31 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:31.784127 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:31.775608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:50019)
0619 02:45:31.776036 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:31.776052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:31.776190 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:31.776521 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:31.776527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:31.776921 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:31.777045 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.777053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.778137 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.778145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.779087 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:31.779096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.779892 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.779897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:31.780046 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.780611 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:31.780627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:31.783803 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
W20260619 02:45:31.784188 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:31.775676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:50019 (local address 127.7.246.126:43749)
0619 02:45:31.775875 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:31.775881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:31.776130 (+ 249us) server_negotiation.cc:407] Connection header received
0619 02:45:31.776319 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:31.776324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:31.776392 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:31.776482 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:31.777197 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.778016 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.779217 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:31.779625 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:31.779783 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:31.783972 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:50019: BlockingRecv error: recv got EOF from 127.7.246.67:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:32.181702 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:32.172124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:56291)
0619 02:45:32.172878 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:32.172906 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:32.173063 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:32.173482 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:32.173488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:32.173934 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:32.174097 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.174106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.175058 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.175063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.176260 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.176273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.176921 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.176925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.177052 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.177968 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:32.177988 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:32.181401 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":244,"threads_started":1}
W20260619 02:45:32.181851 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:32.172234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:56291 (local address 127.7.246.126:43749)
0619 02:45:32.172456 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:32.172463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:32.173017 (+ 554us) server_negotiation.cc:407] Connection header received
0619 02:45:32.173206 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:32.173210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:32.173310 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:32.173409 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:32.174220 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.174915 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.176382 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.176783 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.176869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.181596 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:56291: BlockingRecv error: recv got EOF from 127.7.246.65:56291 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:32.563949 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:32.550460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:52259)
0619 02:45:32.550915 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:32.550930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:32.551077 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:32.551400 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:32.551405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:32.551773 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:32.551941 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.551949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.552910 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.552917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.553925 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.553935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.554630 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.554636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.554808 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.559976 (+ 5168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:32.559993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:32.563714 (+ 3721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
W20260619 02:45:32.564122 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:32.550516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:52259 (local address 127.7.246.126:43749)
0619 02:45:32.550728 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:32.550737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:32.551014 (+ 277us) server_negotiation.cc:407] Connection header received
0619 02:45:32.551194 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:32.551201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:32.551265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:32.551370 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:32.552114 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.552781 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.554066 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.554487 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.554619 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.563879 (+ 9260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:52259: BlockingRecv error: recv got EOF from 127.7.246.66:52259 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:32 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:32.793417 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:32.785306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:47243)
0619 02:45:32.785890 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:32.785910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:32.786016 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:32.786284 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:32.786287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:32.786626 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:32.786777 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.786785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.787715 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.787718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.788620 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:32.788630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.789248 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.789252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:32.789349 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.789985 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:32.790002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:32.793190 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":340,"threads_started":1}
W20260619 02:45:32.793599 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:32.785457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:47243 (local address 127.7.246.126:43749)
0619 02:45:32.785695 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:32.785701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:32.785961 (+ 260us) server_negotiation.cc:407] Connection header received
0619 02:45:32.786103 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:32.786106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:32.786163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:32.786229 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:32.786893 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.787601 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.788730 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:32.789108 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:32.789186 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:32.793362 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:47243: BlockingRecv error: recv got EOF from 127.7.246.67:47243 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:33.191704 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:33.182624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:54549)
0619 02:45:33.183098 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:33.183113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:33.183232 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:33.183517 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:33.183520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:33.183895 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:33.184071 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.184079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.185157 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.185161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.186343 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.186355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.187239 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.187244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.187360 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.188050 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:33.188068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:33.191387 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":305,"threads_started":1}
W20260619 02:45:33.191763 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:33.182727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:54549 (local address 127.7.246.126:43749)
0619 02:45:33.182936 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:33.182940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:33.183195 (+ 255us) server_negotiation.cc:407] Connection header received
0619 02:45:33.183335 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:33.183338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:33.183400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:33.183476 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:33.184200 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.185014 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.186453 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.187073 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.187163 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.191554 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:54549: BlockingRecv error: recv got EOF from 127.7.246.65:54549 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:33.573175 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:33.564884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:50103)
0619 02:45:33.565469 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:33.565484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:33.565617 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:33.565897 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:33.565900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:33.566308 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:33.566474 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.566482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.567478 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.567481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.568362 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.568372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.569044 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.569048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.569170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.569840 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:33.569856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:33.572965 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":348,"threads_started":1}
W20260619 02:45:33.573451 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:33.564949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:50103 (local address 127.7.246.126:43749)
0619 02:45:33.565178 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:33.565185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:33.565596 (+ 411us) server_negotiation.cc:407] Connection header received
0619 02:45:33.565715 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:33.565718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:33.565780 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:33.565860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:33.566612 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.567374 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.568505 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.568905 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.569028 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.573163 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:50103: BlockingRecv error: recv got EOF from 127.7.246.66:50103 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:33 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:33.803097 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:33.794447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:50091)
0619 02:45:33.794860 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:33.794885 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:33.794993 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:33.795268 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:33.795271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:33.795631 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:33.795797 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.795806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.796767 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.796770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.797743 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:33.797753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.798494 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.798500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:33.798682 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.799306 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:33.799322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:33.802747 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":238,"threads_started":1}
W20260619 02:45:33.803210 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:33.794495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:50091 (local address 127.7.246.126:43749)
0619 02:45:33.794717 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:33.794724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:33.794968 (+ 244us) server_negotiation.cc:407] Connection header received
0619 02:45:33.795087 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:33.795091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:33.795158 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:33.795237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:33.795919 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.796661 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.797902 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:33.798346 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:33.798436 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:33.802942 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:50091: BlockingRecv error: recv got EOF from 127.7.246.67:50091 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:34.203382 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:34.192700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:53353)
0619 02:45:34.193320 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:34.193351 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:34.193495 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:34.193947 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:34.193953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:34.194374 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:34.194538 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.194547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.195660 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.195668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.196603 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.196615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.198035 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.198040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.198241 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.198961 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:34.198979 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:34.203023 (+ 4044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":333,"threads_started":1}
W20260619 02:45:34.203485 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:34.192824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:53353 (local address 127.7.246.126:43749)
0619 02:45:34.193054 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:34.193063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:34.193441 (+ 378us) server_negotiation.cc:407] Connection header received
0619 02:45:34.193678 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:34.193682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:34.193796 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:34.193914 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:34.194754 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.195492 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.196747 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.197793 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.197983 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.203235 (+ 5252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:53353: BlockingRecv error: recv got EOF from 127.7.246.65:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:34.584578 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:34.574145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:47029)
0619 02:45:34.574679 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:34.574695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:34.574800 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:34.575070 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:34.575073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:34.575469 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:34.575639 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.575647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.576543 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.576546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.577413 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.577423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.578122 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.578126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.578290 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.579445 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:34.579467 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:34.584214 (+ 4747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":303,"threads_started":1}
W20260619 02:45:34.584681 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:34.574205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:47029 (local address 127.7.246.126:43749)
0619 02:45:34.574359 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:34.574364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:34.574763 (+ 399us) server_negotiation.cc:407] Connection header received
0619 02:45:34.574891 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:34.574895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:34.574954 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:34.575031 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:34.575760 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.576435 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.577521 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.578008 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.578099 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.584416 (+ 6317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:47029: BlockingRecv error: recv got EOF from 127.7.246.66:47029 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:34 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:34.813091 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:34.804010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:51695)
0619 02:45:34.804571 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:34.804597 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:34.804735 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:34.805132 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:34.805137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:34.805777 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:34.805978 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.805987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.807186 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.807190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.808079 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:34.808090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.808671 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.808674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:34.808798 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.809445 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:34.809462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:34.812765 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
W20260619 02:45:34.813233 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:34.804062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:51695 (local address 127.7.246.126:43749)
0619 02:45:34.804286 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:34.804295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:34.804677 (+ 382us) server_negotiation.cc:407] Connection header received
0619 02:45:34.804861 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:34.804867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:34.804972 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:34.805071 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:34.806116 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.807051 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.808205 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:34.808564 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:34.808648 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:34.812983 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:51695: BlockingRecv error: recv got EOF from 127.7.246.67:51695 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:35.213654 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:35.204645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:53577)
0619 02:45:35.205194 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:35.205216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:35.205375 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:35.205851 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:35.205858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:35.206457 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:35.206629 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.206640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.207620 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.207625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.208610 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.208622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.209326 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.209330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.209448 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.210042 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:35.210059 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:35.213364 (+ 3305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
W20260619 02:45:35.213809 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:35.204711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:53577 (local address 127.7.246.126:43749)
0619 02:45:35.204939 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:35.204949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:35.205298 (+ 349us) server_negotiation.cc:407] Connection header received
0619 02:45:35.205503 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:35.205506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:35.205695 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:35.205791 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:35.206762 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.207466 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.208762 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.209205 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.209311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.213605 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:53577: BlockingRecv error: recv got EOF from 127.7.246.65:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.66@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:35.597103 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:35.587292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.66:38657)
0619 02:45:35.588163 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:35.588179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:35.588292 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:35.588631 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:35.588635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:35.589053 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:35.589219 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.589228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.590301 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.590305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.591292 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.591304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.592009 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.592014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.592207 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.592914 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:35.592934 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:35.596801 (+ 3867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":650,"threads_started":1}
W20260619 02:45:35.597199 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:35.587523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:38657 (local address 127.7.246.126:43749)
0619 02:45:35.587805 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:35.587815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:35.588256 (+ 441us) server_negotiation.cc:407] Connection header received
0619 02:45:35.588387 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:35.588391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:35.588452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:35.588535 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:35.589351 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.590145 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.591406 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.591815 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.591903 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.596974 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.66:38657: BlockingRecv error: recv got EOF from 127.7.246.66:38657 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:35 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.67@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:35.822288 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:35.814032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.67:51283)
0619 02:45:35.814483 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:35.814501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:35.814648 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:35.814963 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:35.814967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:35.815370 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:35.815505 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.815512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.816506 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.816512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.817404 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:35.817414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.818107 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.818113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:35.818279 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.818832 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:35.818848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:35.821967 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":261,"threads_started":1}
W20260619 02:45:35.822383 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:35.814072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:51283 (local address 127.7.246.126:43749)
0619 02:45:35.814247 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:35.814255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:35.814585 (+ 330us) server_negotiation.cc:407] Connection header received
0619 02:45:35.814763 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:35.814768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:35.814841 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:35.814936 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:35.815641 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.816374 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.817536 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:35.817970 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:35.818095 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:35.822155 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.67:51283: BlockingRecv error: recv got EOF from 127.7.246.67:51283 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:45:36 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:45:36 dist-test-slave-9tkz krb5kdc[16393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.7.246.65@KRBTEST.COM for oryx/127.7.246.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:45:36.223563 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:36.214935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43749 (local address 127.7.246.65:44611)
0619 02:45:36.215461 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:36.215477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:36.215637 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:36.215961 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:36.215966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:36.216352 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:36.216506 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:36.216514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.217513 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.217519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:36.218449 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:36.218459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.219187 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.219193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:36.219367 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:36.219977 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:36.219993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:36.223214 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.7.246.126:43749: Server oryx/127.7.246.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
W20260619 02:45:36.223637 17399 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:36.214998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:44611 (local address 127.7.246.126:43749)
0619 02:45:36.215270 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:36.215276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:36.215557 (+ 281us) server_negotiation.cc:407] Connection header received
0619 02:45:36.215759 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:36.215764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:36.215844 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:36.215933 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:36.216645 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.217381 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.218582 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.219045 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.219179 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:36.223383 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.246.65:44611: BlockingRecv error: recv got EOF from 127.7.246.65:44611 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260619 02:45:36.496411 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16980
I20260619 02:45:36.509752 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:36.497644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.7.246.65:38187)
0619 02:45:36.498374 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:36.498385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:36.498421 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:45:36.498489 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:36.498496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:36.498742 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:36.498903 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:36.500415 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.501677 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.502817 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.503374 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.503475 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:36.506272 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:36.506281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:36.506295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:36.506345 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:36.509103 (+ 2758us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:45:36.509393 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59302: Request ticket server kudu/127.7.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":389,"threads_started":1}
W20260619 02:45:36.509747 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16980: Not authorized: Client connection negotiation failed: client connection to 127.7.246.65:38187: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 02:45:36.509866 17443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59302: Request ticket server kudu/127.7.246.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:45:36.509968 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16980
I20260619 02:45:36.527259 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17132
I20260619 02:45:36.539675 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:36.528339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.7.246.66:45259)
0619 02:45:36.528991 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:36.529000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:36.529029 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:45:36.529100 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:36.529108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:36.529416 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:36.529650 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:36.530881 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.532014 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.533192 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.533883 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.533981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:36.536726 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:36.536734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:36.536746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:36.536784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:36.539162 (+ 2378us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:45:36.539432 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35154: Request ticket server kudu/127.7.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":322,"threads_started":1}
W20260619 02:45:36.539806 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17132: Not authorized: Client connection negotiation failed: client connection to 127.7.246.66:45259: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 02:45:36.539899 17444 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35154: Request ticket server kudu/127.7.246.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:45:36.539981 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17132
I20260619 02:45:36.557041 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17265
I20260619 02:45:36.569470 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:36.558162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.7.246.67:43365)
0619 02:45:36.558736 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:36.558744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:36.558767 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:36.558828 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:36.558835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:36.559061 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:36.559286 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:36.560606 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.561730 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.563058 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:36.563655 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:36.563801 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:36.566438 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:36.566446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:36.566459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:36.566497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:36.568769 (+ 2272us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:45:36.569144 (+ 375us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39030: Request ticket server kudu/127.7.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
W20260619 02:45:36.569485 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17265: Not authorized: Client connection negotiation failed: client connection to 127.7.246.67:43365: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.7.246.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:45:36.569675 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17265
I20260619 02:45:36.588583 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 16911
I20260619 02:45:36.590015 16973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:36.705225 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 16911
2026-06-19T02:45:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21157 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:36Z Disabled control of system clock
I20260619 02:45:36.778872 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:33571
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:45:37.093029 17450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:37.093353 17450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:37.093462 17450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:37.100648 17450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:37.100749 17450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:37.100793 17450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:37.100848 17450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:37.100890 17450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:37.110014 17450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:33571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:37.111248 17450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:37.112900 17450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:37.124560 17458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:37.124640 17455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:37.124590 17456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:37.125017 17450 server_base.cc:1034] running on GCE node
I20260619 02:45:37.126104 17450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:37.127498 17450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:37.128713 17450 hybrid_clock.cc:648] HybridClock initialized: now 1781837137128667 us; error 67 us; skew 500 ppm
I20260619 02:45:37.129042 17450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:37.131122 17450 webserver.cc:460] Webserver started at http://127.7.246.126:34661/ using document root <none> and password file <none>
I20260619 02:45:37.131793 17450 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:37.131913 17450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:37.132218 17450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:37.135056 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "b5a9e07ce11a4113a0141f8311c061b9"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "c44b828b5effa5a9f62a16cc97b869f2"
server_key_iv: "7d9072d3d6a42794a50dae65ed5a85d5"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.135836 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "b5a9e07ce11a4113a0141f8311c061b9"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "c44b828b5effa5a9f62a16cc97b869f2"
server_key_iv: "7d9072d3d6a42794a50dae65ed5a85d5"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.140589 17450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:45:37.143857 17464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:37.145109 17450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:45:37.145279 17450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "b5a9e07ce11a4113a0141f8311c061b9"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "c44b828b5effa5a9f62a16cc97b869f2"
server_key_iv: "7d9072d3d6a42794a50dae65ed5a85d5"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.145483 17450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:37.164769 17450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:37.165580 17450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:37.165886 17450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:37.182631 17450 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:33571
I20260619 02:45:37.182653 17515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:33571 every 8 connection(s)
I20260619 02:45:37.184028 17450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:37.186602 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17450
I20260619 02:45:37.186830 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:45:37.187600 8153 external_mini_cluster.cc:1428] Setting key ee61a8a174d58f83dc003ce6bd9243d8
I20260619 02:45:37.188526 17516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:37.202327 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:37.189653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.7.246.126:33571)
0619 02:45:37.190685 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:37.190699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:37.190749 (+ 50us) server_negotiation.cc:407] Connection header received
0619 02:45:37.192297 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:37.192324 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:37.192957 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:37.193385 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:37.193933 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.195968 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.197316 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.198351 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.198528 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:37.199572 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:37.199583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:37.199605 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:37.200319 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:37.200417 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:37.200447 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:37.200569 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:37.200733 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 02:45:37.201066 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":319,"threads_started":1}
I20260619 02:45:37.203327 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9: Bootstrap starting.
I20260619 02:45:37.206252 17516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:37.207180 17516 log.cc:824] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:37.209640 17516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9: No bootstrap required, opened a new log
I20260619 02:45:37.216356 17516 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } }
I20260619 02:45:37.216719 17516 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:37.216830 17516 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a9e07ce11a4113a0141f8311c061b9, State: Initialized, Role: FOLLOWER
I20260619 02:45:37.217427 17516 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } }
I20260619 02:45:37.217690 17516 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:37.217834 17516 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:37.218000 17516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:37.220006 17516 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } }
I20260619 02:45:37.220476 17516 leader_election.cc:302] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5a9e07ce11a4113a0141f8311c061b9; no voters:
I20260619 02:45:37.221021 17516 leader_election.cc:288] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:45:37.221153 17521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:37.221835 17521 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [term 1 LEADER]: Becoming Leader. State: Replica: b5a9e07ce11a4113a0141f8311c061b9, State: Running, Role: LEADER
I20260619 02:45:37.222374 17521 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } }
I20260619 02:45:37.223052 17516 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:37.224747 17523 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a9e07ce11a4113a0141f8311c061b9. Latest consensus state: current_term: 1 leader_uuid: "b5a9e07ce11a4113a0141f8311c061b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } } }
I20260619 02:45:37.225082 17523 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:37.226776 17526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:37.226866 17522 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5a9e07ce11a4113a0141f8311c061b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5a9e07ce11a4113a0141f8311c061b9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 33571 } } }
I20260619 02:45:37.227289 17522 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:37.232805 17526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:37.240980 17526 catalog_manager.cc:1347] Generated new cluster ID: 7faf4a707c5d45338466178c820fdb66
I20260619 02:45:37.241086 17526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:37.263430 17526 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:45:37.264663 17526 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:37.273061 17526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a9e07ce11a4113a0141f8311c061b9: Generated new TSK 0
I20260619 02:45:37.273923 17526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:45:37.290385 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--builtin_ntp_servers=127.7.246.84:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:45:37.636168 17541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:37.636435 17541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:37.636520 17541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:37.643726 17541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:37.643842 17541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:37.643970 17541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:37.653358 17541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:37.654660 17541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:37.656445 17541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:37.670285 17546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:37.670228 17549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:37.670265 17547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:37.671244 17541 server_base.cc:1034] running on GCE node
I20260619 02:45:37.671887 17541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:37.672856 17541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:37.674073 17541 hybrid_clock.cc:648] HybridClock initialized: now 1781837137674031 us; error 64 us; skew 500 ppm
I20260619 02:45:37.674393 17541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:37.680863 17541 webserver.cc:460] Webserver started at http://127.7.246.65:33837/ using document root <none> and password file <none>
I20260619 02:45:37.681629 17541 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:37.681752 17541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:37.682089 17541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:37.684885 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "616b3bb1d8a344149aaea92ebbe34ab1"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "ff3a702900a8e07b584a32f023f5954c"
server_key_iv: "8a974083232fc1e9ed8ef8f3a2e83e30"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.685761 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "616b3bb1d8a344149aaea92ebbe34ab1"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "ff3a702900a8e07b584a32f023f5954c"
server_key_iv: "8a974083232fc1e9ed8ef8f3a2e83e30"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.691252 17541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:45:37.694860 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:37.696161 17541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:37.696342 17541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "616b3bb1d8a344149aaea92ebbe34ab1"
format_stamp: "Formatted at 2026-06-19 02:45:37 on dist-test-slave-9tkz"
server_key: "ff3a702900a8e07b584a32f023f5954c"
server_key_iv: "8a974083232fc1e9ed8ef8f3a2e83e30"
server_key_version: "encryptionkey@0"
I20260619 02:45:37.696566 17541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:37.727655 17541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:37.728591 17541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:37.728900 17541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:37.730052 17541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:37.731807 17541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:37.731887 17541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:37.731967 17541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:37.732019 17541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:37.763533 17541 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:38871
I20260619 02:45:37.763618 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:38871 every 8 connection(s)
I20260619 02:45:37.764850 17541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:37.767617 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17541
I20260619 02:45:37.767814 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:45:37.768429 8153 external_mini_cluster.cc:1428] Setting key d5105a032a82ca51726018da09dfbf66
I20260619 02:45:37.770951 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--builtin_ntp_servers=127.7.246.84:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:45:37.777240 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:37.767337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:36423 (local address 127.7.246.126:33571)
0619 02:45:37.767874 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:37.767882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:37.769631 (+ 1749us) server_negotiation.cc:407] Connection header received
0619 02:45:37.770912 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:37.770917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:37.770995 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:37.771099 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:37.772238 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.772942 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.774612 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.775053 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.775209 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:37.776536 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:37.776541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:37.776544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:37.776561 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:37.776607 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:37.776619 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:37.776690 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:37.776915 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:45:37.776981 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":254,"threads_started":1}
I20260619 02:45:37.778453 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:37.768025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:33571 (local address 127.7.246.65:36423)
0619 02:45:37.768993 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:37.769034 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:37.770660 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:37.771329 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:37.771337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:37.771422 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:37.772072 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:37.772098 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.773073 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.773079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:37.774474 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:37.774485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:37.775347 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:37.775353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:37.775553 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:37.776200 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:37.776203 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:37.776239 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:37.776357 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:37.776709 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:37.776718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:37.776728 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:45:37.776905 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 02:45:37.777329 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":313,"threads_started":1}
I20260619 02:45:37.779874 17668 heartbeater.cc:347] Connected to a master server at 127.7.246.126:33571
I20260619 02:45:37.780143 17668 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:37.780817 17668 heartbeater.cc:511] Master 127.7.246.126:33571 requested a full tablet report, sending...
I20260619 02:45:37.782588 17481 ts_manager.cc:194] Registered new tserver with Master: 616b3bb1d8a344149aaea92ebbe34ab1 (127.7.246.65:38871)
I20260619 02:45:37.783952 17481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.65:36423
W20260619 02:45:38.085362 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:38.085692 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:38.085839 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:38.092744 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:38.092877 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:38.093045 17672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:38.102001 17672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:38.103221 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:38.104857 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:38.117374 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:38.117348 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:38.117348 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:38.118186 17672 server_base.cc:1034] running on GCE node
I20260619 02:45:38.118747 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:38.119616 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:38.120838 17672 hybrid_clock.cc:648] HybridClock initialized: now 1781837138120787 us; error 48 us; skew 500 ppm
I20260619 02:45:38.121153 17672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:38.123225 17672 webserver.cc:460] Webserver started at http://127.7.246.66:36005/ using document root <none> and password file <none>
I20260619 02:45:38.123869 17672 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:38.123989 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:38.124306 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:38.126997 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "400827db801c46ad9d66cddd4c2c34dc"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "a32c277523e5ceab3d0a6b1a63006a74"
server_key_iv: "b5c2d94c7c14299e71e73ce9edac28bf"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.127761 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "400827db801c46ad9d66cddd4c2c34dc"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "a32c277523e5ceab3d0a6b1a63006a74"
server_key_iv: "b5c2d94c7c14299e71e73ce9edac28bf"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.132473 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:45:38.135743 17686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.136998 17672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:45:38.137166 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "400827db801c46ad9d66cddd4c2c34dc"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "a32c277523e5ceab3d0a6b1a63006a74"
server_key_iv: "b5c2d94c7c14299e71e73ce9edac28bf"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.137368 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:38.154443 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:38.155200 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:38.155479 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:38.156320 17672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:38.157660 17672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:38.157768 17672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.157904 17672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:38.157994 17672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.189682 17672 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:40655
I20260619 02:45:38.189726 17798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:40655 every 8 connection(s)
I20260619 02:45:38.191052 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:38.201282 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17672
I20260619 02:45:38.201480 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:38.202044 8153 external_mini_cluster.cc:1428] Setting key 89060d5f09cfe48117204130492a405e
I20260619 02:45:38.204428 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--builtin_ntp_servers=127.7.246.84:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:45:38.204521 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.193680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:38069 (local address 127.7.246.126:33571)
0619 02:45:38.193872 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.193878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.195365 (+ 1487us) server_negotiation.cc:407] Connection header received
0619 02:45:38.197739 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.197743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.197815 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.197901 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.199113 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.199964 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.201850 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.202296 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.202434 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.203810 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:38.203816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:38.203821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:38.203843 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:38.203869 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:38.203879 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:38.203962 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:38.204159 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:45:38.204250 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:45:38.205462 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.194194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:33571 (local address 127.7.246.66:38069)
0619 02:45:38.195205 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:38.195245 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:38.197416 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:38.198139 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:38.198150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:38.198260 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:38.198956 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:38.198976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.200103 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.200109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:38.201696 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:38.201712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.202512 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.202518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:38.202807 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.203517 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:38.203519 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:38.203559 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:38.203624 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:38.203961 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:38.203972 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:38.203982 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:45:38.204151 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 02:45:38.204493 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":298,"threads_started":1}
I20260619 02:45:38.206733 17799 heartbeater.cc:347] Connected to a master server at 127.7.246.126:33571
I20260619 02:45:38.207022 17799 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:38.207861 17799 heartbeater.cc:511] Master 127.7.246.126:33571 requested a full tablet report, sending...
I20260619 02:45:38.209249 17481 ts_manager.cc:194] Registered new tserver with Master: 400827db801c46ad9d66cddd4c2c34dc (127.7.246.66:40655)
I20260619 02:45:38.210381 17481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.66:38069
W20260619 02:45:38.516870 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:38.517112 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:38.517249 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:38.524462 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:38.524578 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:38.524735 17802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:38.533727 17802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:38.535069 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:38.537088 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:38.550300 17807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:38.550282 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:38.550191 17810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:38.551164 17802 server_base.cc:1034] running on GCE node
I20260619 02:45:38.551733 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:38.552609 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:38.553818 17802 hybrid_clock.cc:648] HybridClock initialized: now 1781837138553755 us; error 68 us; skew 500 ppm
I20260619 02:45:38.554143 17802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:38.556226 17802 webserver.cc:460] Webserver started at http://127.7.246.67:36549/ using document root <none> and password file <none>
I20260619 02:45:38.556849 17802 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:38.556931 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:38.557197 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:38.559916 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "271cfb075e9444e597670471a7fc6bfd"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "465a481187176109235b133cfffb0c30"
server_key_iv: "7116e25fb66793205055c544c9b9a4d8"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.560621 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "271cfb075e9444e597670471a7fc6bfd"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "465a481187176109235b133cfffb0c30"
server_key_iv: "7116e25fb66793205055c544c9b9a4d8"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.565641 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:45:38.568753 17816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.570083 17802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:38.570276 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "271cfb075e9444e597670471a7fc6bfd"
format_stamp: "Formatted at 2026-06-19 02:45:38 on dist-test-slave-9tkz"
server_key: "465a481187176109235b133cfffb0c30"
server_key_iv: "7116e25fb66793205055c544c9b9a4d8"
server_key_version: "encryptionkey@0"
I20260619 02:45:38.570479 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:38.592304 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:38.593130 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:38.593436 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:38.594364 17802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:38.595769 17802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:38.595888 17802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.596002 17802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:38.596082 17802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:38.628068 17802 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:46601
I20260619 02:45:38.628125 17928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:46601 every 8 connection(s)
I20260619 02:45:38.629379 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:38.635782 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17802
I20260619 02:45:38.636009 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:38.636576 8153 external_mini_cluster.cc:1428] Setting key 6c70623bad3d4b2309713916d5d1261a
I20260619 02:45:38.642014 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.631818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:60015 (local address 127.7.246.126:33571)
0619 02:45:38.632013 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.632018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.633362 (+ 1344us) server_negotiation.cc:407] Connection header received
0619 02:45:38.635119 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.635126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.635215 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.635311 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.636754 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.637644 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.639452 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.639848 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.639983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.641266 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:38.641273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:38.641278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:38.641300 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:38.641336 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:38.641348 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:38.641415 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:38.641670 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:45:38.641739 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:45:38.642930 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.632382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:33571 (local address 127.7.246.67:60015)
0619 02:45:38.633224 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:38.633265 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:38.634871 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:38.635609 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:38.635625 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:38.635741 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:38.636578 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:38.636605 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.637922 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.637926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:38.639319 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:38.639338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.640085 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.640092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:38.640300 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.640944 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:38.640946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:38.640983 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:38.641058 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:38.641432 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:38.641440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:38.641446 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:45:38.641644 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 02:45:38.641920 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":245,"threads_started":1}
I20260619 02:45:38.644301 17929 heartbeater.cc:347] Connected to a master server at 127.7.246.126:33571
I20260619 02:45:38.644580 17929 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:38.645285 17929 heartbeater.cc:511] Master 127.7.246.126:33571 requested a full tablet report, sending...
I20260619 02:45:38.646606 17481 ts_manager.cc:194] Registered new tserver with Master: 271cfb075e9444e597670471a7fc6bfd (127.7.246.67:46601)
I20260619 02:45:38.647325 17481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.67:60015
I20260619 02:45:38.651448 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:45:38.656970 17670 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:38.655659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.7.246.126:33571)
0619 02:45:38.655903 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.655921 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.656168 (+ 247us) server_negotiation.cc:407] Connection header received
0619 02:45:38.656290 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.656294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.656356 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.656427 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.656777 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40724: BlockingRecv error: recv got EOF from 127.0.0.1:40724 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:45:38.657357 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17541
I20260619 02:45:38.664435 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.658274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.7.246.65:38871)
0619 02:45:38.658767 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.658773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.658796 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:38.658852 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.658857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.659030 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.659149 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.659491 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.660465 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.661707 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.662196 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.662285 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.663193 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:38.663200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:38.663210 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:38.663706 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:38.663767 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:38.663784 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:38.663863 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:38.664012 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:45:38.664137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":263,"threads_started":1}
I20260619 02:45:38.665688 17663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:38.746726 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17541
I20260619 02:45:38.763312 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17672
I20260619 02:45:38.770654 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.764206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.7.246.66:40655)
0619 02:45:38.764793 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.764800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.764826 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:45:38.764890 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.764897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.765099 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.765231 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.765653 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.766613 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.767747 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.768261 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.768345 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.769281 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:38.769288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:38.769301 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:38.769973 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:38.770022 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:38.770040 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:38.770122 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:38.770255 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:45:38.770369 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":306,"threads_started":1}
I20260619 02:45:38.771853 17794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:38.851570 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17672
I20260619 02:45:38.867357 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17802
I20260619 02:45:38.874980 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:38.868220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.7.246.67:46601)
0619 02:45:38.868821 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:38.868828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:38.868857 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:45:38.868914 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:38.868920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:38.869131 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:38.869279 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:38.869654 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.870648 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.871873 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:38.872401 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:38.872494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:38.873417 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:38.873426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:38.873439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:38.874247 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:38.874295 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:38.874314 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:38.874398 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:38.874526 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:45:38.874648 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
I20260619 02:45:38.876178 17924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:38.956825 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17802
I20260619 02:45:38.973507 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17450
I20260619 02:45:38.974640 17511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:39.073104 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17450
2026-06-19T02:45:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2362 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:39Z Disabled control of system clock
I20260619 02:45:39.141814 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45155
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:45:39.467504 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:39.467788 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:39.467870 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:39.474812 17948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:39.474918 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:39.474961 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:39.475010 17948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:39.475059 17948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:39.484134 17948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:45155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:39.485332 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:39.487066 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:39.498399 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:39.498834 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:39.498986 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:39.499565 17948 server_base.cc:1034] running on GCE node
I20260619 02:45:39.500308 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:39.501626 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:39.502849 17948 hybrid_clock.cc:648] HybridClock initialized: now 1781837139502802 us; error 56 us; skew 500 ppm
I20260619 02:45:39.503196 17948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:39.505344 17948 webserver.cc:460] Webserver started at http://127.7.246.126:42595/ using document root <none> and password file <none>
I20260619 02:45:39.506070 17948 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:39.506187 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:39.506512 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:39.509338 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "c127f271297c47d39e0016da4c7576c9"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "41820a4d40cd412c1f8517b9b3cd52a9"
server_key_iv: "11bbc4fde749cbbaa7640ba68a570830"
server_key_version: "encryptionkey@0"
I20260619 02:45:39.510164 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "c127f271297c47d39e0016da4c7576c9"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "41820a4d40cd412c1f8517b9b3cd52a9"
server_key_iv: "11bbc4fde749cbbaa7640ba68a570830"
server_key_version: "encryptionkey@0"
I20260619 02:45:39.515007 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:45:39.518332 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:39.519629 17948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:39.519819 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "c127f271297c47d39e0016da4c7576c9"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "41820a4d40cd412c1f8517b9b3cd52a9"
server_key_iv: "11bbc4fde749cbbaa7640ba68a570830"
server_key_version: "encryptionkey@0"
I20260619 02:45:39.520021 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:39.525825 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:39.526563 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:39.526854 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:39.544270 17948 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:45155
I20260619 02:45:39.544308 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:45155 every 8 connection(s)
I20260619 02:45:39.545646 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:39.549160 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:39.550009 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 17948
I20260619 02:45:39.550225 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:45:39.550835 8153 external_mini_cluster.cc:1428] Setting key 6ba820676ae76b0635af3d9399e77883
I20260619 02:45:39.558758 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:39.552814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.7.246.126:45155)
0619 02:45:39.553836 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:39.553846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:39.553891 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:45:39.555335 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:39.555358 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:39.555770 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:39.556096 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:39.557159 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:39.557165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:39.557177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:39.557191 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:39.557250 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:39.557266 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:39.557422 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:39.557622 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:45:39.557906 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":369,"threads_started":1}
I20260619 02:45:39.561758 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9: Bootstrap starting.
I20260619 02:45:39.564483 18014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:39.565462 18014 log.cc:824] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:39.568048 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9: No bootstrap required, opened a new log
I20260619 02:45:39.574719 18014 raft_consensus.cc:348] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } }
I20260619 02:45:39.575088 18014 raft_consensus.cc:374] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:39.575186 18014 raft_consensus.cc:729] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c127f271297c47d39e0016da4c7576c9, State: Initialized, Role: FOLLOWER
I20260619 02:45:39.575759 18014 consensus_queue.cc:260] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } }
I20260619 02:45:39.575973 18014 raft_consensus.cc:388] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:39.576131 18014 raft_consensus.cc:482] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:39.576331 18014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:39.578534 18014 raft_consensus.cc:504] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } }
I20260619 02:45:39.578996 18014 leader_election.cc:302] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c127f271297c47d39e0016da4c7576c9; no voters:
I20260619 02:45:39.579530 18014 leader_election.cc:288] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:45:39.579665 18019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:39.580365 18019 raft_consensus.cc:686] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [term 1 LEADER]: Becoming Leader. State: Replica: c127f271297c47d39e0016da4c7576c9, State: Running, Role: LEADER
I20260619 02:45:39.581014 18019 consensus_queue.cc:237] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } }
I20260619 02:45:39.581533 18014 sys_catalog.cc:564] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:39.583750 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c127f271297c47d39e0016da4c7576c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } } }
I20260619 02:45:39.583755 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c127f271297c47d39e0016da4c7576c9. Latest consensus state: current_term: 1 leader_uuid: "c127f271297c47d39e0016da4c7576c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c127f271297c47d39e0016da4c7576c9" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45155 } } }
I20260619 02:45:39.584168 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:39.584167 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:39.586239 18024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:39.590710 18024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:39.598678 18024 catalog_manager.cc:1347] Generated new cluster ID: 7f02303efdb0480594d7c47affe0f3ae
I20260619 02:45:39.598783 18024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:39.618285 18024 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:45:39.618712 18024 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:39.625111 18024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c127f271297c47d39e0016da4c7576c9: Generated new TSK 0
I20260619 02:45:39.625942 18024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:45:39.636103 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--builtin_ntp_servers=127.7.246.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:45:39.961149 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:39.961410 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:39.961580 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:39.969794 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:39.969919 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:39.970104 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:39.978792 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:39.979983 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:39.981813 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:39.994792 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:39.994868 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:39.994839 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:39.995795 18038 server_base.cc:1034] running on GCE node
I20260619 02:45:39.996343 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:39.997115 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:39.998296 18038 hybrid_clock.cc:648] HybridClock initialized: now 1781837139998261 us; error 33 us; skew 500 ppm
I20260619 02:45:39.998600 18038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:40.000592 18038 webserver.cc:460] Webserver started at http://127.7.246.65:46227/ using document root <none> and password file <none>
I20260619 02:45:40.001219 18038 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:40.001323 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:40.001680 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:40.005860 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "3763a8233788462a8ac60f23f00ce27c"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "aa063f99378d855f2fdca25a235eea50"
server_key_iv: "390e22a54e9334af554cd215a57bb080"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.006623 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "3763a8233788462a8ac60f23f00ce27c"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "aa063f99378d855f2fdca25a235eea50"
server_key_iv: "390e22a54e9334af554cd215a57bb080"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.011561 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:45:40.014775 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.016006 18038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:40.016175 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "3763a8233788462a8ac60f23f00ce27c"
format_stamp: "Formatted at 2026-06-19 02:45:39 on dist-test-slave-9tkz"
server_key: "aa063f99378d855f2fdca25a235eea50"
server_key_iv: "390e22a54e9334af554cd215a57bb080"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.016377 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:40.021783 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:40.022526 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:40.022786 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:40.023670 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:40.025058 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:40.025182 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.025295 18038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:40.025375 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.056541 18038 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36093
I20260619 02:45:40.056592 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36093 every 8 connection(s)
I20260619 02:45:40.057945 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:40.059181 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18038
I20260619 02:45:40.059372 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:45:40.060217 8153 external_mini_cluster.cc:1428] Setting key 802c15b31da7af7505f688700974c07a
I20260619 02:45:40.063647 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--builtin_ntp_servers=127.7.246.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:45:40.067791 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.060775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52761 (local address 127.7.246.126:45155)
0619 02:45:40.061268 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:40.061275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:40.062703 (+ 1428us) server_negotiation.cc:407] Connection header received
0619 02:45:40.065443 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:40.065447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:40.065501 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:40.065644 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:40.067158 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:40.067162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:40.067165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:40.067179 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:40.067216 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:40.067227 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:40.067315 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:40.067538 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 02:45:40.067587 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260619 02:45:40.069125 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.061404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45155 (local address 127.7.246.65:52761)
0619 02:45:40.062507 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:40.062590 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:40.065128 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:40.065893 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:40.065907 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:40.066024 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:40.066771 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:40.066777 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:40.066829 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:40.066967 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:40.067325 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:40.067337 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:40.067348 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:45:40.067639 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 02:45:40.067888 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
I20260619 02:45:40.070611 18165 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45155
I20260619 02:45:40.070874 18165 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:40.071391 18165 heartbeater.cc:511] Master 127.7.246.126:45155 requested a full tablet report, sending...
I20260619 02:45:40.073486 17979 ts_manager.cc:194] Registered new tserver with Master: 3763a8233788462a8ac60f23f00ce27c (127.7.246.65:36093)
W20260619 02:45:40.387154 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:40.387413 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:40.387497 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:40.394425 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:40.394526 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:40.394629 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:40.403149 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:40.404420 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:40.406080 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:40.418390 18177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:40.418411 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:40.418582 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:40.419234 18169 server_base.cc:1034] running on GCE node
I20260619 02:45:40.419818 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:40.420756 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:40.421991 18169 hybrid_clock.cc:648] HybridClock initialized: now 1781837140421944 us; error 54 us; skew 500 ppm
I20260619 02:45:40.422309 18169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:40.424459 18169 webserver.cc:460] Webserver started at http://127.7.246.66:38137/ using document root <none> and password file <none>
I20260619 02:45:40.425077 18169 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:40.425159 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:40.425410 18169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:40.428088 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "5bc47a5d475d4946a8c46a2578089701"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "4ec9f5c31e9bf8d09c2ba24b3d7cbc99"
server_key_iv: "721ce8527b875a4a099c8b180c28c955"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.428781 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "5bc47a5d475d4946a8c46a2578089701"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "4ec9f5c31e9bf8d09c2ba24b3d7cbc99"
server_key_iv: "721ce8527b875a4a099c8b180c28c955"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.433733 18169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:45:40.436913 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.438230 18169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:40.438410 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "5bc47a5d475d4946a8c46a2578089701"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "4ec9f5c31e9bf8d09c2ba24b3d7cbc99"
server_key_iv: "721ce8527b875a4a099c8b180c28c955"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.438622 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:40.444065 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:40.444802 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:40.445056 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:40.445955 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:40.447280 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:40.447397 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.447507 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:40.447590 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.479346 18169 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:34113
I20260619 02:45:40.479393 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:34113 every 8 connection(s)
I20260619 02:45:40.480628 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:40.487789 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18169
I20260619 02:45:40.487915 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:40.488443 8153 external_mini_cluster.cc:1428] Setting key 64e3dfe934b1d2fab6018861175696b3
I20260619 02:45:40.489113 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.483231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:52263 (local address 127.7.246.126:45155)
0619 02:45:40.483393 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:40.483397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:40.484815 (+ 1418us) server_negotiation.cc:407] Connection header received
0619 02:45:40.486495 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:40.486500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:40.486550 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:40.486642 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:40.488381 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:40.488386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:40.488390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:40.488410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:40.488450 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:40.488466 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:40.488584 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:40.488808 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:45:40.488867 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:45:40.490247 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.483794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45155 (local address 127.7.246.66:52263)
0619 02:45:40.484682 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:40.484716 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:40.486251 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:40.486837 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:40.486845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:40.486983 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:40.487878 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:40.487887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:40.487947 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:40.488148 (+ 201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:40.488592 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:40.488605 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:40.488612 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:45:40.488840 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 02:45:40.489111 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260619 02:45:40.491241 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--builtin_ntp_servers=127.7.246.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:45:40.491569 18296 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45155
I20260619 02:45:40.491881 18296 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:40.492579 18296 heartbeater.cc:511] Master 127.7.246.126:45155 requested a full tablet report, sending...
I20260619 02:45:40.494017 17979 ts_manager.cc:194] Registered new tserver with Master: 5bc47a5d475d4946a8c46a2578089701 (127.7.246.66:34113)
W20260619 02:45:40.820977 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:40.821244 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:40.821334 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:40.829026 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:40.829130 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:40.829245 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:40.838289 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:40.839497 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:40.841262 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:40.854843 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:40.854794 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:40.854889 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:40.858927 18299 server_base.cc:1034] running on GCE node
I20260619 02:45:40.859452 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:40.860283 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:40.861469 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781837140861415 us; error 52 us; skew 500 ppm
I20260619 02:45:40.861891 18299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:45:40.864138 18299 webserver.cc:460] Webserver started at http://127.7.246.67:37575/ using document root <none> and password file <none>
I20260619 02:45:40.864780 18299 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:40.864888 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:40.865206 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:40.868013 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "c68f6450da10432e8f44a7dbee9fa9ef"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "63bf33fd5201e218258327f54dfe4a68"
server_key_iv: "31ba211a5894b22c5c7f3dffe5f770b1"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.868750 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "c68f6450da10432e8f44a7dbee9fa9ef"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "63bf33fd5201e218258327f54dfe4a68"
server_key_iv: "31ba211a5894b22c5c7f3dffe5f770b1"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.873513 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 02:45:40.876675 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.877930 18299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:45:40.878084 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "c68f6450da10432e8f44a7dbee9fa9ef"
format_stamp: "Formatted at 2026-06-19 02:45:40 on dist-test-slave-9tkz"
server_key: "63bf33fd5201e218258327f54dfe4a68"
server_key_iv: "31ba211a5894b22c5c7f3dffe5f770b1"
server_key_version: "encryptionkey@0"
I20260619 02:45:40.878293 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:40.883662 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:40.884382 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:40.884629 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:40.885499 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:40.886795 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:40.886931 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.887044 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:40.887132 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:40.918642 18299 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:32925
I20260619 02:45:40.918722 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:32925 every 8 connection(s)
I20260619 02:45:40.919929 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:40.923695 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18299
I20260619 02:45:40.923861 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:40.924463 8153 external_mini_cluster.cc:1428] Setting key 499519d7782bc8320fa90ddf67d46042
I20260619 02:45:40.928560 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.922459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:59207 (local address 127.7.246.126:45155)
0619 02:45:40.922651 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:40.922656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:40.924236 (+ 1580us) server_negotiation.cc:407] Connection header received
0619 02:45:40.926371 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:40.926376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:40.926447 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:40.926546 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:40.927858 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:40.927863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:40.927872 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:40.927887 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:40.927915 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:40.927925 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:40.928058 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:40.928246 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:45:40.928308 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:45:40.929420 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.922986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45155 (local address 127.7.246.67:59207)
0619 02:45:40.924086 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:40.924138 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:40.926084 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:40.926771 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:40.926779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:40.926877 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:40.927596 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:45:40.927598 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:40.927632 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:45:40.927709 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:40.928039 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:40.928049 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:40.928057 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:45:40.928319 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 02:45:40.928561 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":316,"threads_started":1}
I20260619 02:45:40.930557 18426 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45155
I20260619 02:45:40.930860 18426 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:40.931409 18426 heartbeater.cc:511] Master 127.7.246.126:45155 requested a full tablet report, sending...
I20260619 02:45:40.932631 17979 ts_manager.cc:194] Registered new tserver with Master: c68f6450da10432e8f44a7dbee9fa9ef (127.7.246.67:32925)
I20260619 02:45:40.939154 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:45:40.944905 18167 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:45:40.943466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.7.246.126:45155)
0619 02:45:40.943636 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:40.943644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:40.944133 (+ 489us) server_negotiation.cc:407] Connection header received
0619 02:45:40.944251 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:40.944254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:40.944299 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:40.944374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:40.944674 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51882: BlockingRecv error: recv got EOF from 127.0.0.1:51882 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:45:40.945684 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18038
I20260619 02:45:40.948756 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:40.946503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.7.246.65:36093)
0619 02:45:40.946977 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:40.946983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:40.947006 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:45:40.947055 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:40.947061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:40.947249 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:40.947374 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:40.948191 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:40.948197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:40.948207 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:40.948235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:40.948276 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:40.948289 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:40.948385 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:40.948459 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:45:40.948529 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260619 02:45:40.949844 18160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:41.029688 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18038
I20260619 02:45:41.045675 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18169
I20260619 02:45:41.049029 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:41.046554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.7.246.66:34113)
0619 02:45:41.047031 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:41.047038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:41.047063 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:45:41.047126 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:41.047132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:41.047340 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:41.047460 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:41.048283 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:41.048289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:41.048301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:41.048318 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:41.048357 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:41.048370 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:41.048497 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:41.048573 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:45:41.048677 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260619 02:45:41.050410 18291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:41.129894 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18169
I20260619 02:45:41.145263 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18299
I20260619 02:45:41.149098 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:41.146239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.7.246.67:32925)
0619 02:45:41.146819 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:41.146826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:41.146854 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:45:41.146912 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:41.146918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:41.147127 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:41.147295 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:41.148385 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:41.148391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:41.148402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:45:41.148418 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:41.148455 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:45:41.148468 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:41.148639 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:41.148711 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:45:41.148806 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260619 02:45:41.150300 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:41.229274 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18299
I20260619 02:45:41.244938 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 17948
I20260619 02:45:41.246068 18009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:41.342057 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 17948
2026-06-19T02:45:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2268 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:45:41 dist-test-slave-9tkz krb5kdc[18444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:45:41 dist-test-slave-9tkz krb5kdc[18444](info): set up 2 sockets
Jun 19 02:45:41 dist-test-slave-9tkz krb5kdc[18444](info): commencing operation
krb5kdc: starting...
W20260619 02:45:42.422537 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:45:42 dist-test-slave-9tkz krb5kdc[18444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:45:42.651367 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:42049
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:42.963392 18457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:42.963690 18457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:42.963841 18457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:42.970674 18457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:45:42.970801 18457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:42.970898 18457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:42.970984 18457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:45:42.971061 18457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:45:42.981686 18457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:42.982931 18457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:42.984197 18457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:42.995396 18462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:42.995425 18463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:42.995483 18465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:42.996351 18457 server_base.cc:1034] running on GCE node
I20260619 02:45:42.997015 18457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:42.998387 18457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:42.999611 18457 hybrid_clock.cc:648] HybridClock initialized: now 1781837142999563 us; error 61 us; skew 500 ppm
I20260619 02:45:42.999935 18457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:43 dist-test-slave-9tkz krb5kdc[18444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:43.006193 18457 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:45:43.007637 18457 webserver.cc:460] Webserver started at http://127.7.246.126:41199/ using document root <none> and password file <none>
I20260619 02:45:43.008273 18457 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:43.008405 18457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:43.008678 18457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:43.011459 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "7d44853ba21e469db98ffd1b5ed8a360"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "4459f6e31b950820c8b3374f829303fc"
server_key_iv: "eca4141f49162a3db624ca2693acecf0"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.012176 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "7d44853ba21e469db98ffd1b5ed8a360"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "4459f6e31b950820c8b3374f829303fc"
server_key_iv: "eca4141f49162a3db624ca2693acecf0"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.017028 18457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:45:43.020160 18472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:43.021353 18457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:45:43.021508 18457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "7d44853ba21e469db98ffd1b5ed8a360"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "4459f6e31b950820c8b3374f829303fc"
server_key_iv: "eca4141f49162a3db624ca2693acecf0"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.021725 18457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:43.034539 18457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:43.038508 18457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:43.038763 18457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:43.055732 18457 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:42049
I20260619 02:45:43.055763 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:42049 every 8 connection(s)
I20260619 02:45:43.057001 18457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:45:43.060402 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:43.061319 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18457
I20260619 02:45:43.061517 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:45:43.062137 8153 external_mini_cluster.cc:1428] Setting key 6e73dcc931bf220ae2991d65a8b929d6
I20260619 02:45:43.073678 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: Bootstrap starting.
Jun 19 02:45:43 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:43.077040 18524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:43.078097 18524 log.cc:824] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:43.080946 18524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: No bootstrap required, opened a new log
I20260619 02:45:43.084410 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:43.063873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.7.246.126:42049)
0619 02:45:43.064568 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:43.064578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:43.064616 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:45:43.066193 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:43.066232 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:43.066568 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:43.066864 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:43.068679 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.069941 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.071737 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.072379 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.072502 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:43.076847 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:43.076854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:43.076865 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:43.076894 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:43.080328 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:43.080953 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:43.080963 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:43.080972 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:43.081167 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:43.081488 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:43.081493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:43.081497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:43.082161 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:43.082436 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:43.082716 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:45:43.083118 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
I20260619 02:45:43.090185 18524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } }
I20260619 02:45:43.090500 18524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:43.090580 18524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d44853ba21e469db98ffd1b5ed8a360, State: Initialized, Role: FOLLOWER
I20260619 02:45:43.091038 18524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } }
I20260619 02:45:43.091228 18524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:45:43.091322 18524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:45:43.091434 18524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:43.093410 18524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } }
I20260619 02:45:43.093861 18524 leader_election.cc:302] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d44853ba21e469db98ffd1b5ed8a360; no voters:
I20260619 02:45:43.094362 18524 leader_election.cc:288] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:45:43.094480 18529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:43.095157 18529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [term 1 LEADER]: Becoming Leader. State: Replica: 7d44853ba21e469db98ffd1b5ed8a360, State: Running, Role: LEADER
I20260619 02:45:43.095651 18529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } }
I20260619 02:45:43.096520 18524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:45:43.098479 18530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d44853ba21e469db98ffd1b5ed8a360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } } }
I20260619 02:45:43.098892 18530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:43.101764 18531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d44853ba21e469db98ffd1b5ed8a360. Latest consensus state: current_term: 1 leader_uuid: "7d44853ba21e469db98ffd1b5ed8a360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d44853ba21e469db98ffd1b5ed8a360" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42049 } } }
I20260619 02:45:43.102151 18537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:45:43.102154 18531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360 [sys.catalog]: This master's current role is: LEADER
I20260619 02:45:43.107208 18537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:45:43.114739 18537 catalog_manager.cc:1347] Generated new cluster ID: c6e029d88b1442da9529589ccdbfb593
I20260619 02:45:43.114845 18537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:45:43.146361 18537 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:45:43.147684 18537 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:45:43.163842 18537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: Generated new TSK 0
I20260619 02:45:43.164738 18537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:45:43.216763 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:43.532358 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:43.532613 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:43.532799 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:43.540614 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:43.540737 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:43.540895 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:45:43.550239 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:43.551440 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:43.552757 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:43.565579 18555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:43.565676 18556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:43.565518 18558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:43.566838 18550 server_base.cc:1034] running on GCE node
I20260619 02:45:43.567457 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:43.568331 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:43.569607 18550 hybrid_clock.cc:648] HybridClock initialized: now 1781837143569553 us; error 44 us; skew 500 ppm
I20260619 02:45:43.569862 18550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:43 dist-test-slave-9tkz krb5kdc[18444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837143, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:43.572847 18550 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:45:43.574277 18550 webserver.cc:460] Webserver started at http://127.7.246.65:36869/ using document root <none> and password file <none>
I20260619 02:45:43.574929 18550 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:43.575095 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:43.575393 18550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:43.578385 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "d38677577e054393aef0aadd5b45d206"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "ea3e40ba26c4a3187f7c5a5a48381b9a"
server_key_iv: "a062ad1a95dcba32a4f692736aed1b3e"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.579210 18550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "d38677577e054393aef0aadd5b45d206"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "ea3e40ba26c4a3187f7c5a5a48381b9a"
server_key_iv: "a062ad1a95dcba32a4f692736aed1b3e"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.585919 18550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 02:45:43.591552 18565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:43.593148 18550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:45:43.593307 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "d38677577e054393aef0aadd5b45d206"
format_stamp: "Formatted at 2026-06-19 02:45:43 on dist-test-slave-9tkz"
server_key: "ea3e40ba26c4a3187f7c5a5a48381b9a"
server_key_iv: "a062ad1a95dcba32a4f692736aed1b3e"
server_key_version: "encryptionkey@0"
I20260619 02:45:43.593577 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:43.615115 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:43.619059 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:43.619328 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:43.620223 18550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:43.621515 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:43.621662 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:43.621803 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:43.621874 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:43.653878 18550 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:42641
I20260619 02:45:43.653923 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:42641 every 8 connection(s)
I20260619 02:45:43.655262 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:45:43.658201 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18550
I20260619 02:45:43.658399 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:45:43.659243 8153 external_mini_cluster.cc:1428] Setting key c0146a900cee893255567070621231b0
Jun 19 02:45:43 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837143, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:43.676658 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:43.657896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:51653 (local address 127.7.246.126:42049)
0619 02:45:43.658367 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:43.658373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:43.659610 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 02:45:43.661988 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:43.661993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:43.662084 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:43.662190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:43.663868 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.664681 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.666398 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.666830 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.666950 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:43.670969 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:43.670974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:43.670977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:43.671005 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:43.673390 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:43.674392 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:43.674395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:43.674398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:43.674515 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:43.675003 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:43.675006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:43.675007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:43.675268 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:43.675367 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:43.676276 (+ 909us) server_negotiation.cc:299] Negotiation successful
0619 02:45:43.676428 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":274,"threads_started":1}
I20260619 02:45:43.678447 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:43.658639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42049 (local address 127.7.246.65:51653)
0619 02:45:43.659451 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:43.659493 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:43.661719 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:43.662434 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:43.662442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:43.663072 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:43.663724 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:43.663742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.664819 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.664824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:43.666258 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:43.666283 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:43.667060 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:43.667067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:43.667282 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:43.667976 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:43.668001 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:43.670645 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:43.673526 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:43.673539 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:43.673594 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:43.674274 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:43.674619 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:43.674627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:43.674634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:43.674892 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:43.675394 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:43.675405 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:43.675861 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 02:45:43.676211 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 02:45:43.676807 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":317,"threads_started":1}
I20260619 02:45:43.680066 18678 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42049
I20260619 02:45:43.680392 18678 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:43.681085 18678 heartbeater.cc:511] Master 127.7.246.126:42049 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:45:43.683184 18489 ts_manager.cc:194] Registered new tserver with Master: d38677577e054393aef0aadd5b45d206 (127.7.246.65:42641)
I20260619 02:45:43.684749 18489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:51653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:45:43.712435 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:44.032382 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:44.032675 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:44.032859 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:44.040150 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:44.040277 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:44.040469 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:45:44.050421 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:44.051719 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:44.053179 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:44.066681 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:44.066815 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:44.067402 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:44.068114 18684 server_base.cc:1034] running on GCE node
I20260619 02:45:44.068871 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:44.070020 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:44.071254 18684 hybrid_clock.cc:648] HybridClock initialized: now 1781837144071204 us; error 38 us; skew 500 ppm
I20260619 02:45:44.071607 18684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:44 dist-test-slave-9tkz krb5kdc[18444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837144, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:44.075286 18684 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:45:44.076642 18684 webserver.cc:460] Webserver started at http://127.7.246.66:44147/ using document root <none> and password file <none>
I20260619 02:45:44.077310 18684 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:44.077445 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:44.077792 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:44.080747 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "2d3afded3c0e4347b0fe09ef5dabb84c"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "e0eac8dcf5e6ab5a80fea780d96f5ef8"
server_key_iv: "7d0360831cb90304e3a69068b9152dbb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.081528 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "2d3afded3c0e4347b0fe09ef5dabb84c"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "e0eac8dcf5e6ab5a80fea780d96f5ef8"
server_key_iv: "7d0360831cb90304e3a69068b9152dbb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.086755 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 02:45:44.090296 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:44.091678 18684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:45:44.091845 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "2d3afded3c0e4347b0fe09ef5dabb84c"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "e0eac8dcf5e6ab5a80fea780d96f5ef8"
server_key_iv: "7d0360831cb90304e3a69068b9152dbb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.092038 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:44.111732 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:44.115710 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:44.115995 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:44.116930 18684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:44.118350 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:44.118417 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:44.118510 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:44.118552 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:44.151257 18684 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:32837
I20260619 02:45:44.151295 18811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:32837 every 8 connection(s)
I20260619 02:45:44.152549 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:45:44.154208 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18684
I20260619 02:45:44.154386 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:45:44.154830 8153 external_mini_cluster.cc:1428] Setting key cac0e2f6dfcc8170aad48daaf34574d2
Jun 19 02:45:44 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837144, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:44.173976 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:49371 (local address 127.7.246.126:42049)
0619 02:45:44.155521 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.155525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.156910 (+ 1385us) server_negotiation.cc:407] Connection header received
0619 02:45:44.158843 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.158852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.158944 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.159053 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:44.160641 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.161379 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.163073 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.163822 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.163970 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.168206 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:44.168211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:44.168215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:44.168247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:44.170905 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.171649 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.171655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.171663 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.171780 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.172214 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.172218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.172221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.172482 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:44.172652 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.173344 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.173606 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:45:44.175026 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.155917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42049 (local address 127.7.246.66:49371)
0619 02:45:44.156757 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.156813 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.158541 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.159263 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.159271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.159803 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:44.160438 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.160456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.161602 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.161606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.162900 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.162912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.164107 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.164124 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.164426 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.165147 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:44.165189 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:44.167962 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:44.171062 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:44.171076 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:44.171094 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:44.171517 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:44.171895 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:44.171900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:44.171904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:44.172097 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:44.172627 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:44.172635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:44.172985 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.173264 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.173849 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":336,"threads_started":1}
I20260619 02:45:44.176599 18812 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42049
I20260619 02:45:44.176893 18812 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:45:44.177712 18812 heartbeater.cc:511] Master 127.7.246.126:42049 requested a full tablet report, sending...
I20260619 02:45:44.179133 18489 ts_manager.cc:194] Registered new tserver with Master: 2d3afded3c0e4347b0fe09ef5dabb84c (127.7.246.66:32837)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:45:44.180315 18489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:49371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:45:44.208647 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:45:44.536826 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:45:44.537128 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:45:44.537302 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:45:44.545051 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:45:44.545177 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:45:44.545389 18817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:45:44.555184 18817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:45:44.556525 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:45:44.557981 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:45:44.571597 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:44.571576 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:45:44.571888 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:45:44.572666 18817 server_base.cc:1034] running on GCE node
I20260619 02:45:44.573318 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:45:44.574260 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:45:44.575500 18817 hybrid_clock.cc:648] HybridClock initialized: now 1781837144575446 us; error 45 us; skew 500 ppm
I20260619 02:45:44.575806 18817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:45:44 dist-test-slave-9tkz krb5kdc[18444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837144, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:45:44.578773 18817 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:45:44.580119 18817 webserver.cc:460] Webserver started at http://127.7.246.67:42643/ using document root <none> and password file <none>
I20260619 02:45:44.580773 18817 fs_manager.cc:359] Metadata directory not provided
I20260619 02:45:44.580921 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:45:44.581209 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:45:44.585561 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "3f58a14666f24760a23e022794c15be0"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "2506423996f15e399b3f2fa96f80ca41"
server_key_iv: "e3a33a6d2e7a7e19634a905b4b30fcdb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.586566 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "3f58a14666f24760a23e022794c15be0"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "2506423996f15e399b3f2fa96f80ca41"
server_key_iv: "e3a33a6d2e7a7e19634a905b4b30fcdb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.593252 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 02:45:44.597043 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:44.598434 18817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:45:44.598587 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "3f58a14666f24760a23e022794c15be0"
format_stamp: "Formatted at 2026-06-19 02:45:44 on dist-test-slave-9tkz"
server_key: "2506423996f15e399b3f2fa96f80ca41"
server_key_iv: "e3a33a6d2e7a7e19634a905b4b30fcdb"
server_key_version: "encryptionkey@0"
I20260619 02:45:44.598763 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:45:44.618571 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:45:44.622679 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:45:44.622913 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:45:44.623816 18817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:45:44.625356 18817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:45:44.625432 18817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:45:44.625516 18817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:45:44.625607 18817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 02:45:44.659317 18817 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:43231
I20260619 02:45:44.659375 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:43231 every 8 connection(s)
I20260619 02:45:44.660661 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:45:44.669831 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 18817
I20260619 02:45:44.670049 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:45:44.670483 8153 external_mini_cluster.cc:1428] Setting key 0f2c6813bcdb7413b115058345aae06b
Jun 19 02:45:44 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837144, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:45:44.682029 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.663642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:53323 (local address 127.7.246.126:42049)
0619 02:45:44.663835 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.663841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.665020 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 02:45:44.666779 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.666784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.666849 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.666927 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:44.668346 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.669089 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.670808 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.671593 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.671760 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.675886 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:44.675894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:44.675900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:44.675942 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:44.678792 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.679541 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.679546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.679549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.679836 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.680317 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.680322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.680326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.680593 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:44.680740 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.681429 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.681666 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:45:44.682972 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.664186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42049 (local address 127.7.246.67:53323)
0619 02:45:44.664871 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.664911 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.666526 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.667115 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.667122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.667612 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:45:44.668145 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.668161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.669233 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.669236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.670665 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.670677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.671990 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.671999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.672218 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.673035 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:45:44.673066 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:45:44.675600 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:45:44.678980 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:44.678989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:44.679006 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:44.679416 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:44.679963 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:45:44.679970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:45:44.679974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:45:44.680173 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:45:44.680730 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:45:44.680739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:45:44.681073 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.681330 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.681898 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":279,"threads_started":1}
I20260619 02:45:44.684476 18945 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42049
I20260619 02:45:44.684856 18945 heartbeater.cc:464] Registering TS with master...
I20260619 02:45:44.685940 18945 heartbeater.cc:511] Master 127.7.246.126:42049 requested a full tablet report, sending...
I20260619 02:45:44.687338 18489 ts_manager.cc:194] Registered new tserver with Master: 3f58a14666f24760a23e022794c15be0 (127.7.246.67:43231)
I20260619 02:45:44.687440 18678 heartbeater.cc:503] Master 127.7.246.126:42049 was elected leader, sending a full tablet report...
I20260619 02:45:44.688325 18489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:53323
I20260619 02:45:44.698307 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:45:44.716383 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.703133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.7.246.126:42049)
0619 02:45:44.703361 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.703370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.703813 (+ 443us) server_negotiation.cc:407] Connection header received
0619 02:45:44.704014 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.704019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.704121 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.704227 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:44.705605 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.706350 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.707553 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.707989 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.708119 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.711182 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:44.711187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:44.711191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:44.711226 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:44.713925 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.714621 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.714626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.714628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.714763 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.715140 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.715144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.715146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.715471 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:44.715656 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.715975 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.716151 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 02:45:44.720487 18489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:45:44.727254 18489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:45:44.761932 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.739390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.7.246.65:42641)
0619 02:45:44.753082 (+ 13692us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.753089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.753113 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:45:44.753229 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.753236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.753504 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.753767 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:44.754329 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.755936 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.758941 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.760572 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.760716 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.760971 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.761311 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.761495 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13547,"thread_start_us":13404,"threads_started":1}
I20260619 02:45:44.762271 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.738973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:42641 (local address 127.0.0.1:48102)
0619 02:45:44.751882 (+ 12909us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.751944 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.752176 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.753767 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.753774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.753805 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:44.754170 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.754179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.756082 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.756088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.758769 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.758786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.760789 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.760796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.761039 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.761099 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.761249 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.761425 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12555,"mutex_wait_us":39,"thread_start_us":14428,"threads_started":1}
I20260619 02:45:44.762737 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.739198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:32837 (local address 127.0.0.1:44166)
0619 02:45:44.753706 (+ 14508us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.753727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.753864 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.754909 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.754914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.754943 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:44.755536 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.755551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.757310 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.757315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.759590 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.759604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.761973 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.761982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.762160 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.762193 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.762298 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.762429 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14361,"thread_start_us":12206,"threads_started":1}
I20260619 02:45:44.762960 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.739389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.7.246.66:32837)
0619 02:45:44.751702 (+ 12313us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.751712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.754031 (+ 2319us) server_negotiation.cc:407] Connection header received
0619 02:45:44.754158 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.754176 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.754437 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.754666 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:44.755759 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.757124 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.759788 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.761172 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.761293 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.761467 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.762480 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.762637 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12020,"thread_start_us":12967,"threads_started":1}
I20260619 02:45:44.764030 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.754181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.7.246.67:43231)
0619 02:45:44.754836 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.754844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.755567 (+ 723us) server_negotiation.cc:407] Connection header received
0619 02:45:44.755808 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.755815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.756034 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.756211 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:44.757271 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.758413 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.761215 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.762682 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.762817 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.763055 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.763582 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.763746 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":365,"threads_started":1}
I20260619 02:45:44.764077 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.754868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:43231 (local address 127.0.0.1:58288)
0619 02:45:44.755465 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.755484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.755658 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.756505 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.756509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.756530 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:44.757077 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.757094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.758590 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.758595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.761057 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.761069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.763191 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.763197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.763369 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.763415 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.763538 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.763670 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":125,"thread_start_us":297,"threads_started":1}
I20260619 02:45:44.766661 18613 tablet_service.cc:1467] Processing CreateTablet for tablet ec68c5ee61df4f878792452e944eec9a (DEFAULT_TABLE table=test-table [id=da789e4ac2194bb69271482956b34d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:44.766662 18747 tablet_service.cc:1467] Processing CreateTablet for tablet ec68c5ee61df4f878792452e944eec9a (DEFAULT_TABLE table=test-table [id=da789e4ac2194bb69271482956b34d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:44.767951 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec68c5ee61df4f878792452e944eec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:44.768119 18613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec68c5ee61df4f878792452e944eec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:44.770131 18880 tablet_service.cc:1467] Processing CreateTablet for tablet ec68c5ee61df4f878792452e944eec9a (DEFAULT_TABLE table=test-table [id=da789e4ac2194bb69271482956b34d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:45:44.771553 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec68c5ee61df4f878792452e944eec9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:45:44.778787 18963 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Bootstrap starting.
I20260619 02:45:44.782984 18964 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Bootstrap starting.
I20260619 02:45:44.783454 18963 tablet_bootstrap.cc:654] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:44.784812 18963 log.cc:824] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:44.786257 18964 tablet_bootstrap.cc:654] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:44.787088 18966 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Bootstrap starting.
I20260619 02:45:44.787454 18964 log.cc:824] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:44.788396 18963 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: No bootstrap required, opened a new log
I20260619 02:45:44.788662 18963 ts_tablet_manager.cc:1397] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260619 02:45:44.790539 18966 tablet_bootstrap.cc:654] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Neither blocks nor log segments found. Creating new log.
I20260619 02:45:44.790553 18964 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: No bootstrap required, opened a new log
I20260619 02:45:44.790899 18964 ts_tablet_manager.cc:1397] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 02:45:44.791764 18966 log.cc:824] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:45:44.794510 18966 tablet_bootstrap.cc:492] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: No bootstrap required, opened a new log
I20260619 02:45:44.794764 18966 ts_tablet_manager.cc:1397] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 02:45:44.799891 18963 raft_consensus.cc:348] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.800359 18963 raft_consensus.cc:374] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:44.800454 18963 raft_consensus.cc:729] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d3afded3c0e4347b0fe09ef5dabb84c, State: Initialized, Role: FOLLOWER
I20260619 02:45:44.801069 18963 consensus_queue.cc:260] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.801923 18964 raft_consensus.cc:348] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.802561 18964 raft_consensus.cc:374] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:44.802672 18964 raft_consensus.cc:729] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38677577e054393aef0aadd5b45d206, State: Initialized, Role: FOLLOWER
I20260619 02:45:44.803185 18966 raft_consensus.cc:348] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.803601 18966 raft_consensus.cc:374] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:45:44.803524 18964 consensus_queue.cc:260] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.803757 18966 raft_consensus.cc:729] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f58a14666f24760a23e022794c15be0, State: Initialized, Role: FOLLOWER
I20260619 02:45:44.804302 18966 consensus_queue.cc:260] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.805467 18945 heartbeater.cc:503] Master 127.7.246.126:42049 was elected leader, sending a full tablet report...
I20260619 02:45:44.806034 18966 ts_tablet_manager.cc:1428] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260619 02:45:44.808135 18812 heartbeater.cc:503] Master 127.7.246.126:42049 was elected leader, sending a full tablet report...
I20260619 02:45:44.809024 18963 ts_tablet_manager.cc:1428] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Time spent starting tablet: real 0.020s user 0.017s sys 0.003s
I20260619 02:45:44.810495 18964 ts_tablet_manager.cc:1428] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20260619 02:45:44.862349 18971 raft_consensus.cc:482] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:45:44.862597 18971 raft_consensus.cc:504] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.864133 18971 leader_election.cc:288] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38677577e054393aef0aadd5b45d206 (127.7.246.65:42641), 2d3afded3c0e4347b0fe09ef5dabb84c (127.7.246.66:32837)
I20260619 02:45:44.871239 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.864806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:42157 (local address 127.7.246.66:32837)
0619 02:45:44.865032 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.865039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.865067 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:45:44.865142 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.865150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.865245 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.865384 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:44.865808 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.866808 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.869220 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.870429 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.870595 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.870669 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.870883 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.870998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:45:44.871299 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.864404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:32837 (local address 127.7.246.67:42157)
0619 02:45:44.864646 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.864667 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.864802 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.865404 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.865410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.865437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:44.865663 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.865676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.866939 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.866943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.869062 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.869076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.870562 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.870566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.870693 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.870717 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.870837 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.870979 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":72}
I20260619 02:45:44.871541 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.864600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:42607 (local address 127.7.246.65:42641)
0619 02:45:44.864799 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.864805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.865307 (+ 502us) server_negotiation.cc:407] Connection header received
0619 02:45:44.865454 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.865459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.865560 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.865668 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:45:44.866172 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.867120 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.869400 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.870658 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.870822 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.870896 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.871170 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.871311 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 02:45:44.871907 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.864525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:42641 (local address 127.7.246.67:42607)
0619 02:45:44.865203 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 02:45:44.865215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:45:44.865307 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:45:44.865700 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:45:44.865707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:45:44.865736 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:45:44.866011 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.866018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.867255 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.867264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.869272 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:45:44.869281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.870862 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.870867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:45:44.871018 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.871046 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 02:45:44.871129 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:45:44.871244 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":443,"threads_started":1}
I20260619 02:45:44.872394 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec68c5ee61df4f878792452e944eec9a" candidate_uuid: "3f58a14666f24760a23e022794c15be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" is_pre_election: true
I20260619 02:45:44.872989 18767 raft_consensus.cc:2454] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f58a14666f24760a23e022794c15be0 in term 0.
I20260619 02:45:44.872917 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec68c5ee61df4f878792452e944eec9a" candidate_uuid: "3f58a14666f24760a23e022794c15be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38677577e054393aef0aadd5b45d206" is_pre_election: true
I20260619 02:45:44.873452 18633 raft_consensus.cc:2454] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f58a14666f24760a23e022794c15be0 in term 0.
I20260619 02:45:44.873750 18836 leader_election.cc:302] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d3afded3c0e4347b0fe09ef5dabb84c, 3f58a14666f24760a23e022794c15be0; no voters:
I20260619 02:45:44.874148 18971 raft_consensus.cc:2781] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:45:44.874303 18971 raft_consensus.cc:482] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:45:44.874429 18971 raft_consensus.cc:3037] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:44.876703 18971 raft_consensus.cc:504] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.877302 18971 leader_election.cc:288] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [CANDIDATE]: Term 1 election: Requested vote from peers d38677577e054393aef0aadd5b45d206 (127.7.246.65:42641), 2d3afded3c0e4347b0fe09ef5dabb84c (127.7.246.66:32837)
I20260619 02:45:44.877758 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec68c5ee61df4f878792452e944eec9a" candidate_uuid: "3f58a14666f24760a23e022794c15be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38677577e054393aef0aadd5b45d206"
I20260619 02:45:44.877854 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec68c5ee61df4f878792452e944eec9a" candidate_uuid: "3f58a14666f24760a23e022794c15be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c"
I20260619 02:45:44.877967 18633 raft_consensus.cc:3037] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:44.878046 18767 raft_consensus.cc:3037] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:45:44.881178 18767 raft_consensus.cc:2454] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f58a14666f24760a23e022794c15be0 in term 1.
I20260619 02:45:44.881175 18633 raft_consensus.cc:2454] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f58a14666f24760a23e022794c15be0 in term 1.
I20260619 02:45:44.881600 18833 leader_election.cc:302] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f58a14666f24760a23e022794c15be0, d38677577e054393aef0aadd5b45d206; no voters:
I20260619 02:45:44.881950 18971 raft_consensus.cc:2781] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:45:44.882382 18971 raft_consensus.cc:686] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f58a14666f24760a23e022794c15be0, State: Running, Role: LEADER
I20260619 02:45:44.882926 18971 consensus_queue.cc:237] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } }
I20260619 02:45:44.887329 18488 catalog_manager.cc:5495] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f58a14666f24760a23e022794c15be0 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "3f58a14666f24760a23e022794c15be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f58a14666f24760a23e022794c15be0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43231 } health_report { overall_health: HEALTHY } } }
W20260619 02:45:44.905861 18813 tablet.cc:2369] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:45:44.907485 18679 tablet.cc:2369] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:45:44.912375 18946 tablet.cc:2369] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:45:44.916855 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.911905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.7.246.67:43231)
0619 02:45:44.912093 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.912097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.912136 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:45:44.912292 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.912297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.912408 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.912564 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:45:44.912963 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.913724 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.915048 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.915444 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.915530 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.915934 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:45:44.916391 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:45:44.916497 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.916607 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.916666 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:45:44.924729 18633 raft_consensus.cc:1261] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 1 FOLLOWER]: Refusing update from remote peer 3f58a14666f24760a23e022794c15be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:44.924801 18767 raft_consensus.cc:1261] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 1 FOLLOWER]: Refusing update from remote peer 3f58a14666f24760a23e022794c15be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:45:44.925729 18973 consensus_queue.cc:1035] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38677577e054393aef0aadd5b45d206" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 42641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:44.926097 18971 consensus_queue.cc:1035] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d3afded3c0e4347b0fe09ef5dabb84c" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:45:44.938422 18977 mvcc.cc:205] Tried to move back new op lower bound from 7298404945601966080 to 7298404945444507648. Current Snapshot: MvccSnapshot[applied={T|T < 7298404945601966080}]
I20260619 02:45:44.955262 18488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:44.955545 18488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:45:44.959379 18488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d44853ba21e469db98ffd1b5ed8a360: Sending DeleteTablet for 3 replicas of tablet ec68c5ee61df4f878792452e944eec9a
I20260619 02:45:44.960592 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet ec68c5ee61df4f878792452e944eec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:44 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:44166
I20260619 02:45:44.960613 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet ec68c5ee61df4f878792452e944eec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:44 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:58288
I20260619 02:45:44.960613 18613 tablet_service.cc:1514] Processing DeleteTablet for tablet ec68c5ee61df4f878792452e944eec9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:45:44 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:48102
I20260619 02:45:44.963704 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18550
I20260619 02:45:44.975063 18988 tablet_replica.cc:330] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: stopping tablet replica
I20260619 02:45:44.975657 18988 raft_consensus.cc:2229] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:44.976310 18988 raft_consensus.cc:2258] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:44.977213 18987 tablet_replica.cc:330] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: stopping tablet replica
I20260619 02:45:44.977766 18987 raft_consensus.cc:2229] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:45:44.978322 18987 raft_consensus.cc:2258] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:44.979379 18986 tablet_replica.cc:330] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: stopping tablet replica
I20260619 02:45:44.979753 18986 raft_consensus.cc:2229] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:45:44.980520 18986 raft_consensus.cc:2258] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:45:44.982049 18988 ts_tablet_manager.cc:1905] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 02:45:44 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:45:44.985392 18987 ts_tablet_manager.cc:1905] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:44.986629 18986 ts_tablet_manager.cc:1905] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:45:44.988857 18988 ts_tablet_manager.cc:1918] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:44.989017 18988 log.cc:1196] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/ec68c5ee61df4f878792452e944eec9a
I20260619 02:45:44.989529 18988 ts_tablet_manager.cc:1939] T ec68c5ee61df4f878792452e944eec9a P 2d3afded3c0e4347b0fe09ef5dabb84c: Deleting consensus metadata
I20260619 02:45:44.991379 18476 catalog_manager.cc:4841] TS 2d3afded3c0e4347b0fe09ef5dabb84c (127.7.246.66:32837): tablet ec68c5ee61df4f878792452e944eec9a (table test-table [id=da789e4ac2194bb69271482956b34d6b]) successfully deleted
I20260619 02:45:44.993727 18986 ts_tablet_manager.cc:1918] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:44.993959 18986 log.cc:1196] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/ec68c5ee61df4f878792452e944eec9a
I20260619 02:45:44.994606 18986 ts_tablet_manager.cc:1939] T ec68c5ee61df4f878792452e944eec9a P 3f58a14666f24760a23e022794c15be0: Deleting consensus metadata
I20260619 02:45:44.996330 18473 catalog_manager.cc:4841] TS 3f58a14666f24760a23e022794c15be0 (127.7.246.67:43231): tablet ec68c5ee61df4f878792452e944eec9a (table test-table [id=da789e4ac2194bb69271482956b34d6b]) successfully deleted
I20260619 02:45:44.996486 18987 ts_tablet_manager.cc:1918] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:45:44.996652 18987 log.cc:1196] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/ec68c5ee61df4f878792452e944eec9a
I20260619 02:45:44.997172 18987 ts_tablet_manager.cc:1939] T ec68c5ee61df4f878792452e944eec9a P d38677577e054393aef0aadd5b45d206: Deleting consensus metadata
I20260619 02:45:44.998374 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:44.965015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.7.246.65:42641)
0619 02:45:44.965233 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:44.965243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:44.971273 (+ 6030us) server_negotiation.cc:407] Connection header received
0619 02:45:44.972088 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:44.972094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:44.972186 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:44.972263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:44.975624 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.976804 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.979753 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:44.980414 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:44.980508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:44.990726 (+ 10218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:44.990741 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:44.990758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:44.990802 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:44.994613 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.995556 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.995565 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.995569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.995748 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:44.996202 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:44.996207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:44.996210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:44.996758 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:44.997072 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:44.997709 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 02:45:44.997957 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:45:44.998970 18673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:45.000109 18473 catalog_manager.cc:4841] TS d38677577e054393aef0aadd5b45d206 (127.7.246.65:42641): tablet ec68c5ee61df4f878792452e944eec9a (table test-table [id=da789e4ac2194bb69271482956b34d6b]) successfully deleted
I20260619 02:45:45.088853 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18550
I20260619 02:45:45.112573 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18684
Jun 19 02:45:45 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:45:45.129679 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:45.113366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.7.246.66:32837)
0619 02:45:45.113580 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:45.113587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:45.113609 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:45:45.113774 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:45.113782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:45.113875 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:45.113979 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:45.115612 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:45.116317 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:45.118295 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:45.118708 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:45.118813 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:45.123800 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:45.123813 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:45.123829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:45.123867 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:45.126836 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:45.127514 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:45.127522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:45.127524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:45.127687 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:45.128073 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:45.128080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:45.128087 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:45.128442 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:45.128745 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:45.129071 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 02:45:45.129308 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:45:45.130376 18807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:45.220770 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18684
I20260619 02:45:45.237821 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18817
Jun 19 02:45:45 dist-test-slave-9tkz krb5kdc[18444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:45:45.254937 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:45:45.238747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.7.246.67:43231)
0619 02:45:45.238944 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:45:45.238953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:45:45.238982 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:45:45.239061 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:45:45.239066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:45:45.239149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:45:45.239284 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:45:45.240455 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:45.241851 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:45.243067 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:45:45.243733 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:45:45.243875 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:45:45.248096 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:45:45.248110 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:45:45.248141 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:45:45.248187 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:45:45.252203 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:45.252843 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:45.252853 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:45.252857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:45.253020 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:45:45.253383 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:45:45.253388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:45:45.253392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:45:45.253846 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:45:45.254129 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 02:45:45.254328 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:45:45.254540 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:45:45.255782 18940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:45.375797 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18817
I20260619 02:45:45.394971 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 18457
I20260619 02:45:45.396148 18519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:45:45.502795 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 18457
2026-06-19T02:45:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4162 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:45:45 dist-test-slave-9tkz krb5kdc[18996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:45:45 dist-test-slave-9tkz krb5kdc[18996](info): set up 2 sockets
Jun 19 02:45:45 dist-test-slave-9tkz krb5kdc[18996](info): commencing operation
krb5kdc: starting...
W20260619 02:45:46.594633 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:45:46 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:45:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangeradmin_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangeradmin_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangerlookup_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangerlookup_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
I20260619 02:45:46.919970 8153 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfqHoa6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/postgres -l logfile start
2026-06-19 02:45:49.331 UTC [19026] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:45:49.331 UTC [19026] LOG: listening on IPv4 address "127.7.246.84", port 50029
2026-06-19 02:45:49.335 UTC [19026] LOG: listening on Unix socket "/tmp/.s.PGSQL.50029"
2026-06-19 02:45:49.354 UTC [19029] LOG: database system was shut down at 2026-06-19 02:45:47 UTC
2026-06-19 02:45:49.358 UTC [19026] LOG: database system is ready to accept connections
I20260619 02:45:50.322294 8153 mini_postgres.cc:96] Postgres bound to 50029
2026-06-19 02:45:50.339 UTC [19037] FATAL: database "slave" does not exist
127.7.246.84:50029 - accepting connections
I20260619 02:45:50.339890 8153 mini_ranger.cc:161] Starting Ranger...
I20260619 02:45:50.363855 8153 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:45:50.815112 8153 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:45:50.815281 8153 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin
2026-06-19 02:45:51,108 [I] DB FLAVOR :POSTGRES
2026-06-19 02:45:51,109 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:45:51,109 [I] Checking connection..
2026-06-19 02:45:51,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:45:51,367 [I] Checking connection passed.
2026-06-19 02:45:51,367 [I] --------- Verifying version history table ---------
2026-06-19 02:45:51,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:45:51,633 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:45:51,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:45:51,892 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:45:51,893 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:45:51,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:45:52.145 UTC [19101] WARNING: there is no transaction in progress
2026-06-19 02:45:52,151 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:45:52,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:45:52,406 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:45:52,406 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:45:52,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:45:52,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:45:52,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkz.c.gcp-upstream.internal','N') ;"
2026-06-19 02:45:53,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:45:53,366 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:45:53,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:45:53,602 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:45:53,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:45:53,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:45:54,076 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:45:54,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:45:55.005 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.023 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.039 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.054 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.068 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.085 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.140 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.147 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.161 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.172 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.182 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.196 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.206 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.212 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.217 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55.224 UTC [19209] WARNING: there is no transaction in progress
2026-06-19 02:45:55,825 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:45:55,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:45:56,070 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:45:56,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:45:56,321 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:45:56,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:45:56,547 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:45:56,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:45:56,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:45:57,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkz.c.gcp-upstream.internal';"
2026-06-19 02:45:57,267 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:45:57,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:45:57,528 [I] DB_PATCHES have already been applied
I20260619 02:45:57.534296 8153 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfqHoa6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:45:57.534446 8153 mini_ranger.cc:193] Using host: 127.7.246.84
I20260619 02:45:57.549508 8153 mini_ranger.cc:227] Ranger admin URL: http://127.7.246.84:38845
Jun 19, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:45:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:45:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:45:58.675841 8153 mini_ranger.cc:160] Time spent starting Ranger: real 8.336s user 0.000s sys 0.035s
Jun 19, 2026 2:46:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:46:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:46:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:46:19 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:46:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:46:20.637965 8153 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangerkms_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/rangerkms_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:46:20 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:46:20.801862 8153 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:46:20.823200 8153 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:46:21.639309 8153 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:46:21.639439 8153 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms
I20260619 02:46:21.639592 8153 mini_ranger_kms.cc:227] Using postgres at 127.7.246.84:50029
2026-06-19 02:46:22,047 [I] DB FLAVOR :POSTGRES
2026-06-19 02:46:22,047 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:46:22,047 [I] Checking connection
2026-06-19 02:46:22,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:46:22,305 [I] connection success
2026-06-19 02:46:22,306 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:46:22,306 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:46:22,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:46:22,546 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:46:22,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:46:22,546 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:46:22,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:50029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:46:22,820 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:46:23.313666 8153 mini_ranger_kms.cc:310] Created kms service
I20260619 02:46:23.493558 8153 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:46:23.542840 8153 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:46:23 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837180, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:46:23.943087 8153 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:46:23.943390 8153 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:46:23.943472 8153 mini_ranger_kms.cc:240] Using host: 127.7.246.84
I20260619 02:46:23.965692 8153 mini_ranger_kms.cc:276] Ranger KMS PID: 19390
I20260619 02:46:23.965889 8153 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.246.84:35639
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 02:46:26 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837186, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:46:26.344364 8153 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.543s user 0.104s sys 0.323s
I20260619 02:46:26.344602 8153 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:46:26.344810 8153 mini_ranger_kms.cc:401] 127.7.246.84:35639/kms/v1/keys
Jun 19 02:46:26 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837186, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
Jun 19 02:46:27 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837186, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
Jun 19 02:46:29 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:46:29.301111 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41785
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41785
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:35639/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:46:29.638164 19441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:46:29.638427 19441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:46:29.638520 19441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:46:29.645766 19441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:46:29.645862 19441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:46:29.645906 19441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:46:29.645958 19441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:46:29.646003 19441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:46:29.656227 19441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.246.84:35639/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41785
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:46:29.658215 19441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:46:29.660454 19441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:46:29.676618 19447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:29.676640 19446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:29.677477 19441 server_base.cc:1034] running on GCE node
W20260619 02:46:29.677561 19449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:29.678709 19441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:46:29.680696 19441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:46:29.681947 19441 hybrid_clock.cc:648] HybridClock initialized: now 1781837189681896 us; error 43 us; skew 500 ppm
I20260619 02:46:29.682382 19441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:46:29 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:46:29.686880 19441 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:46:29.688876 19441 webserver.cc:460] Webserver started at http://127.7.246.126:33101/ using document root <none> and password file <none>
I20260619 02:46:29.689782 19441 fs_manager.cc:359] Metadata directory not provided
I20260619 02:46:29.689911 19441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:46:29.690282 19441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:46:29 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:46:29.753293 19441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "f950289a417342bd96fd54f4baa01a45"
format_stamp: "Formatted at 2026-06-19 02:46:29 on dist-test-slave-9tkz"
server_key: "17d4613285c84981d8346f42df830a7b"
server_key_iv: "4c9dc211506e035e834398503e32b139"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:29.754191 19441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "f950289a417342bd96fd54f4baa01a45"
format_stamp: "Formatted at 2026-06-19 02:46:29 on dist-test-slave-9tkz"
server_key: "17d4613285c84981d8346f42df830a7b"
server_key_iv: "4c9dc211506e035e834398503e32b139"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:29.759579 19441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:46:29.775049 19456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:29.776564 19441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:46:29.776734 19441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "f950289a417342bd96fd54f4baa01a45"
format_stamp: "Formatted at 2026-06-19 02:46:29 on dist-test-slave-9tkz"
server_key: "17d4613285c84981d8346f42df830a7b"
server_key_iv: "4c9dc211506e035e834398503e32b139"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:29.776963 19441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:46:29.797230 19441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:46:29.801159 19441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:46:29.801496 19441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:46:29.822636 19441 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41785
I20260619 02:46:29.822650 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41785 every 8 connection(s)
I20260619 02:46:29.824007 19441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:46:29.827839 19508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:46:29.833858 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 19441
I20260619 02:46:29.834053 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:46:29.834842 8153 external_mini_cluster.cc:1428] Setting key 3dfe4b18afe263abf21e4568f5a92051
I20260619 02:46:29.839609 19508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: Bootstrap starting.
I20260619 02:46:29.844915 19508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: Neither blocks nor log segments found. Creating new log.
I20260619 02:46:29.846446 19508 log.cc:824] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: Log is configured to *not* fsync() on all Append() calls
I20260619 02:46:29.853935 19508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: No bootstrap required, opened a new log
I20260619 02:46:29.863037 19508 raft_consensus.cc:348] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } }
I20260619 02:46:29.863502 19508 raft_consensus.cc:374] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:46:29.863613 19508 raft_consensus.cc:729] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f950289a417342bd96fd54f4baa01a45, State: Initialized, Role: FOLLOWER
I20260619 02:46:29.864284 19508 consensus_queue.cc:260] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } }
I20260619 02:46:29.864562 19508 raft_consensus.cc:388] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:46:29.864709 19508 raft_consensus.cc:482] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:46:29.864910 19508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:46:29 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:46:29.867373 19508 raft_consensus.cc:504] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } }
I20260619 02:46:29.867959 19508 leader_election.cc:302] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f950289a417342bd96fd54f4baa01a45; no voters:
I20260619 02:46:29.868671 19508 leader_election.cc:288] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:46:29.868851 19513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:46:29.872927 19513 raft_consensus.cc:686] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [term 1 LEADER]: Becoming Leader. State: Replica: f950289a417342bd96fd54f4baa01a45, State: Running, Role: LEADER
I20260619 02:46:29.873697 19513 consensus_queue.cc:237] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } }
I20260619 02:46:29.877579 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:29.836941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.7.246.126:41785)
0619 02:46:29.851513 (+ 14572us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:29.851528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:29.852053 (+ 525us) server_negotiation.cc:407] Connection header received
0619 02:46:29.854601 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:29.854630 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:29.855106 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:29.855599 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:29.857084 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:29.859326 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:29.861573 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:29.862668 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:29.862844 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:29.867855 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:29.867867 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:29.867885 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:29.867955 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:29.871880 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:29.872663 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:29.872671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:29.872679 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:29.872829 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:29.873257 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:29.873262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:29.873266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:29.874096 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:29.875468 (+ 1372us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:29.875770 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:46:29.876235 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14315,"thread_start_us":14106,"threads_started":1}
I20260619 02:46:29.878412 19508 sys_catalog.cc:564] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:46:29.880535 19514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f950289a417342bd96fd54f4baa01a45. Latest consensus state: current_term: 1 leader_uuid: "f950289a417342bd96fd54f4baa01a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } } }
I20260619 02:46:29.880537 19515 sys_catalog.cc:455] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f950289a417342bd96fd54f4baa01a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f950289a417342bd96fd54f4baa01a45" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41785 } } }
I20260619 02:46:29.880918 19514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [sys.catalog]: This master's current role is: LEADER
I20260619 02:46:29.880919 19515 sys_catalog.cc:458] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45 [sys.catalog]: This master's current role is: LEADER
I20260619 02:46:29.884425 19518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:46:29.889931 19518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:46:29.899714 19508 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:46:29.900164 19518 catalog_manager.cc:1347] Generated new cluster ID: be509d464a554b1a8bb0a1d676bd49a4
I20260619 02:46:29.900280 19518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:46:29.920794 19518 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:46:29.922683 19518 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:46:29.932952 19518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: Generated new TSK 0
I20260619 02:46:29.934031 19518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:46:30.649173 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:46:30 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837190, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:46:31.076186 19553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:46:31.076553 19553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:46:31.076704 19553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:46:31.088953 19553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:46:31.089097 19553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:46:31.089288 19553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:46:31.104601 19553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.246.84:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:46:31.106705 19553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:46:31.108379 19553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:46:31.126374 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:31.127538 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:31.135051 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:31.135483 19553 server_base.cc:1034] running on GCE node
I20260619 02:46:31.136133 19553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:46:31.137162 19553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:46:31.138378 19553 hybrid_clock.cc:648] HybridClock initialized: now 1781837191138318 us; error 55 us; skew 500 ppm
I20260619 02:46:31.138707 19553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:46:31 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837191, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:46:31.141992 19553 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:46:31.144222 19553 webserver.cc:460] Webserver started at http://127.7.246.65:45369/ using document root <none> and password file <none>
I20260619 02:46:31.144953 19553 fs_manager.cc:359] Metadata directory not provided
I20260619 02:46:31.145102 19553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:46:31.145438 19553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:46:31 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837191, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:46:31.168728 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "35f3974ca4194e4a8b070204c2e473da"
format_stamp: "Formatted at 2026-06-19 02:46:31 on dist-test-slave-9tkz"
server_key: "e5d29371620c93292d2177ca897ffc71"
server_key_iv: "f64605facdceb8a3e00c7b5f3852a4cd"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:31.170184 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "35f3974ca4194e4a8b070204c2e473da"
format_stamp: "Formatted at 2026-06-19 02:46:31 on dist-test-slave-9tkz"
server_key: "e5d29371620c93292d2177ca897ffc71"
server_key_iv: "f64605facdceb8a3e00c7b5f3852a4cd"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:31.176963 19553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 02:46:31.210203 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:31.212810 19553 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.005s
I20260619 02:46:31.213070 19553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "35f3974ca4194e4a8b070204c2e473da"
format_stamp: "Formatted at 2026-06-19 02:46:31 on dist-test-slave-9tkz"
server_key: "e5d29371620c93292d2177ca897ffc71"
server_key_iv: "f64605facdceb8a3e00c7b5f3852a4cd"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:31.213434 19553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:46:31.235625 19553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:46:31.239599 19553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:46:31.239897 19553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:46:31.240926 19553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:46:31.242726 19553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:46:31.242830 19553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:31.242973 19553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:46:31.243044 19553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:31.362227 19553 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:40905
I20260619 02:46:31.362283 19685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:40905 every 8 connection(s)
I20260619 02:46:31.363839 19553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:46:31.365972 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 19553
I20260619 02:46:31.366155 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:46:31.366918 8153 external_mini_cluster.cc:1428] Setting key cff8b95b4826b903070b5de0a355d65b
Jun 19 02:46:31 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837191, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:46:31.405860 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:31.369365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:55061 (local address 127.7.246.126:41785)
0619 02:46:31.386191 (+ 16826us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:31.386216 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:31.386237 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:46:31.386296 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:31.386304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:31.386409 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:31.386529 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:31.388660 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:31.390203 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:31.392662 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:31.393340 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:31.393438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:31.397974 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:31.397982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:31.397987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:31.398027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:31.401790 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:31.402690 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:31.402696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:31.402700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:31.402837 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:31.403529 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:31.403534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:31.403538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:31.403873 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:31.404005 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:31.405056 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 02:46:31.405285 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16657,"thread_start_us":16548,"threads_started":1}
I20260619 02:46:31.407557 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:31.369826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41785 (local address 127.7.246.65:55061)
0619 02:46:31.382262 (+ 12436us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:31.382339 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:31.385368 (+ 3029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:31.386965 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:31.386977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:31.387615 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:46:31.388432 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:31.388459 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:31.390409 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:31.390414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:31.392493 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:31.392509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:31.393824 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:31.393832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:31.394118 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:31.394930 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:46:31.394961 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:46:31.397693 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:46:31.401948 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:31.401958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:31.401976 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:31.402556 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:31.402973 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:31.402976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:31.402979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:31.403228 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:31.404118 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:46:31.404126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:46:31.404607 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 02:46:31.405630 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0619 02:46:31.406173 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12012,"thread_start_us":12157,"threads_started":1}
I20260619 02:46:31.411381 19686 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41785
I20260619 02:46:31.411772 19686 heartbeater.cc:464] Registering TS with master...
I20260619 02:46:31.412740 19686 heartbeater.cc:511] Master 127.7.246.126:41785 requested a full tablet report, sending...
I20260619 02:46:31.415438 19473 ts_manager.cc:194] Registered new tserver with Master: 35f3974ca4194e4a8b070204c2e473da (127.7.246.65:40905)
I20260619 02:46:31.417686 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:55061
Jun 19 02:46:31 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837190, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:46:31.469236 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:46:31 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837190, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
W20260619 02:46:31.927167 19692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:46:31.927608 19692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:46:31.927810 19692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:46:31.939469 19692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:46:31.939638 19692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:46:31.939838 19692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:46:31.954888 19692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.246.84:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:46:31.956668 19692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:46:31.958503 19692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:46:31.991164 19700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:31.994730 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:32.001922 19692 server_base.cc:1034] running on GCE node
W20260619 02:46:32.002684 19702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:32.003502 19692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:46:32.006639 19692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:46:32.007957 19692 hybrid_clock.cc:648] HybridClock initialized: now 1781837192007840 us; error 107 us; skew 500 ppm
I20260619 02:46:32.008389 19692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:46:32.012617 19692 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:46:32.019541 19692 webserver.cc:460] Webserver started at http://127.7.246.66:45869/ using document root <none> and password file <none>
I20260619 02:46:32.020602 19692 fs_manager.cc:359] Metadata directory not provided
I20260619 02:46:32.020780 19692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:46:32.021175 19692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:46:32.041255 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "2f78e5ec48584be0a6c481d2de80b86e"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "aa00aa31d5aa0b9a9b423503616bb422"
server_key_iv: "1b58a0226c4e1ab02522d91ac9898519"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.042281 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "2f78e5ec48584be0a6c481d2de80b86e"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "aa00aa31d5aa0b9a9b423503616bb422"
server_key_iv: "1b58a0226c4e1ab02522d91ac9898519"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.059569 19692 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.014s sys 0.001s
I20260619 02:46:32.084506 19711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:32.086704 19692 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.000s
I20260619 02:46:32.086931 19692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "2f78e5ec48584be0a6c481d2de80b86e"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "aa00aa31d5aa0b9a9b423503616bb422"
server_key_iv: "1b58a0226c4e1ab02522d91ac9898519"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.087209 19692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:46:32.122407 19692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:46:32.127105 19692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:46:32.127408 19692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:46:32.128410 19692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:46:32.130045 19692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:46:32.130125 19692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:32.130658 19692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:46:32.130716 19692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:46:32.179836 19692 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:44221
I20260619 02:46:32.179874 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:44221 every 8 connection(s)
I20260619 02:46:32.181535 19692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:46:32.187788 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 19692
I20260619 02:46:32.187958 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:46:32.188515 8153 external_mini_cluster.cc:1428] Setting key 802a801bff8021b0b1681f294b419e08
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:46:32.213526 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.185833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:50953 (local address 127.7.246.126:41785)
0619 02:46:32.187779 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.187785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.194364 (+ 6579us) server_negotiation.cc:407] Connection header received
0619 02:46:32.194424 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.194430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.194530 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.194634 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:32.197892 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.199032 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.201348 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.201964 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.202056 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.206341 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:32.206347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:32.206351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:32.206390 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:32.209873 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.210745 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.210751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.210755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.210887 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.211364 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.211369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.211372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.211746 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:32.211882 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.212839 (+ 957us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.213062 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":4943,"threads_started":1}
I20260619 02:46:32.215996 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.186983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41785 (local address 127.7.246.66:50953)
0619 02:46:32.190948 (+ 3965us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:32.190997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:32.193962 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:32.196056 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:32.196069 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:32.196850 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:46:32.197666 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.197692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.199234 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.199239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.201193 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.201207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.202361 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.202368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.202658 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.203433 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:46:32.203464 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:46:32.206105 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:46:32.210043 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:32.210053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:32.210071 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:32.210615 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:32.211014 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:32.211018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:32.211022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:32.211238 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:32.211964 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:46:32.211972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:46:32.212410 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 02:46:32.214076 (+ 1666us) client_negotiation.cc:240] Negotiation successful
0619 02:46:32.214626 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3666,"thread_start_us":8790,"threads_started":1}
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:46:32.217974 19829 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41785
I20260619 02:46:32.218279 19829 heartbeater.cc:464] Registering TS with master...
I20260619 02:46:32.219075 19829 heartbeater.cc:511] Master 127.7.246.126:41785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:46:32.220806 19473 ts_manager.cc:194] Registered new tserver with Master: 2f78e5ec48584be0a6c481d2de80b86e (127.7.246.66:44221)
I20260619 02:46:32.221885 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:50953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:46:32.253414 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:46:32.421497 19686 heartbeater.cc:503] Master 127.7.246.126:41785 was elected leader, sending a full tablet report...
W20260619 02:46:32.645983 19835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:46:32.646250 19835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:46:32.646343 19835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:46:32.653710 19835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:46:32.653816 19835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:46:32.653925 19835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:46:32.663422 19835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.246.84:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:46:32.664561 19835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:46:32.665820 19835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:46:32.678710 19843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:32.678740 19840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:46:32.678772 19841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:46:32.679582 19835 server_base.cc:1034] running on GCE node
I20260619 02:46:32.680234 19835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:46:32.681173 19835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:46:32.682411 19835 hybrid_clock.cc:648] HybridClock initialized: now 1781837192682370 us; error 50 us; skew 500 ppm
I20260619 02:46:32.682753 19835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:46:32.686164 19835 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:46:32.687645 19835 webserver.cc:460] Webserver started at http://127.7.246.67:35093/ using document root <none> and password file <none>
I20260619 02:46:32.688297 19835 fs_manager.cc:359] Metadata directory not provided
I20260619 02:46:32.688452 19835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:46:32.688812 19835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:46:32.706885 19835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "241a8a9886b34b179dd00a27c23ef6f0"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "cdd0c0ca20598619c993790be7fd3caf"
server_key_iv: "7ccb7cd429087ca7f1b12009cb240ba3"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.707878 19835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "241a8a9886b34b179dd00a27c23ef6f0"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "cdd0c0ca20598619c993790be7fd3caf"
server_key_iv: "7ccb7cd429087ca7f1b12009cb240ba3"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.714569 19835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 02:46:32.732245 19850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:32.733809 19835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:46:32.734010 19835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "241a8a9886b34b179dd00a27c23ef6f0"
format_stamp: "Formatted at 2026-06-19 02:46:32 on dist-test-slave-9tkz"
server_key: "cdd0c0ca20598619c993790be7fd3caf"
server_key_iv: "7ccb7cd429087ca7f1b12009cb240ba3"
server_key_version: "kuduclusterkey@0"
I20260619 02:46:32.734226 19835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:46:32.758744 19835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:46:32.762395 19835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:46:32.762677 19835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:46:32.763567 19835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:46:32.764873 19835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:46:32.764974 19835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:32.765116 19835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:46:32.765185 19835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:46:32.799409 19835 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:34143
I20260619 02:46:32.799444 19962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:34143 every 8 connection(s)
I20260619 02:46:32.800737 19835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:46:32.803467 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 19835
I20260619 02:46:32.803706 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:46:32.804199 8153 external_mini_cluster.cc:1428] Setting key e7faeae00a73ac33e3b95321cdd71685
Jun 19 02:46:32 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837192, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:46:32.820683 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.803292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:40297 (local address 127.7.246.126:41785)
0619 02:46:32.804014 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.804021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.805235 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 02:46:32.807574 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.807578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.807686 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.807796 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:32.809160 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.809933 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.811381 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.811805 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.811930 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.815140 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:32.815144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:32.815147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:32.815175 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:32.817738 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.818569 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.818573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.818575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.818751 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.819226 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.819232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.819234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.819548 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:32.819657 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.820298 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.820448 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":473,"threads_started":1}
I20260619 02:46:32.821635 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.804248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41785 (local address 127.7.246.67:40297)
0619 02:46:32.805093 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:32.805141 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:32.807312 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:32.808037 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:32.808046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:32.808441 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:46:32.809005 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.809022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.810085 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.810090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.811247 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.811258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.812031 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.812038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.812269 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.813006 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:46:32.813061 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:46:32.814903 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:46:32.817885 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:32.817893 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:32.817907 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:32.818446 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:32.818866 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:46:32.818870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:46:32.818874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:46:32.819078 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:46:32.819655 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:46:32.819660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:46:32.820003 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 02:46:32.820247 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 02:46:32.820567 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":351,"threads_started":1}
I20260619 02:46:32.823269 19963 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41785
I20260619 02:46:32.823632 19963 heartbeater.cc:464] Registering TS with master...
I20260619 02:46:32.824285 19963 heartbeater.cc:511] Master 127.7.246.126:41785 requested a full tablet report, sending...
I20260619 02:46:32.825646 19473 ts_manager.cc:194] Registered new tserver with Master: 241a8a9886b34b179dd00a27c23ef6f0 (127.7.246.67:34143)
I20260619 02:46:32.826393 19473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:40297
I20260619 02:46:32.831524 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:46:32.848567 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.835678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.7.246.126:41785)
0619 02:46:32.835808 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.835812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.836301 (+ 489us) server_negotiation.cc:407] Connection header received
0619 02:46:32.836498 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.836502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.836568 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.836688 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:32.838214 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.838934 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.840316 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.840741 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.840878 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.843208 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:32.843212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:32.843216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:32.843246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:32.846041 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.846659 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.846665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.846669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.846794 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:32.847169 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:32.847182 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:32.847185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:32.847433 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:32.847670 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.847999 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.848228 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:46:32.852401 19473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:46:32.859441 19473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:46:32.884696 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.871596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.7.246.65:40905)
0619 02:46:32.872410 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.872420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.873697 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 02:46:32.874318 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.874329 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.874573 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.874729 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:46:32.875820 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.877873 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.881579 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.883216 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.883354 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.883595 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.884163 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.884331 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":507,"threads_started":1}
I20260619 02:46:32.884699 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.871290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:40905 (local address 127.0.0.1:60308)
0619 02:46:32.873549 (+ 2259us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:32.873602 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:32.874115 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:32.875021 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:32.875028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:32.875063 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:46:32.875588 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.875603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.878035 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.878040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.881304 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.881324 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.883727 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.883733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.883927 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.883976 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 02:46:32.884097 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 02:46:32.884291 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1989,"thread_start_us":1885,"threads_started":1}
I20260619 02:46:32.889307 19621 tablet_service.cc:1467] Processing CreateTablet for tablet 397de3a702904b0daf198f10e4655374 (DEFAULT_TABLE table=test-table [id=ab1461c8e288475ca7eec50121d57191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:46:32.890969 19621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397de3a702904b0daf198f10e4655374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:46:32.891419 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.871727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:34143 (local address 127.0.0.1:57182)
0619 02:46:32.873582 (+ 1855us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:32.873601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:32.873776 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:32.884587 (+ 10811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:32.884591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:32.884608 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:46:32.885389 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.885406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.887313 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.887321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.888981 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.888996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.890598 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.890603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.890772 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.890803 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:46:32.890894 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:46:32.891038 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"spinlock_wait_cycles":768,"thread_start_us":1387,"threads_started":1}
I20260619 02:46:32.891443 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.872182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.7.246.67:34143)
0619 02:46:32.884015 (+ 11833us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.884021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.884048 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:46:32.884158 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.884165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.884395 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.884670 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:46:32.885723 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.887124 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.889160 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.890409 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.890549 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.890759 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.890962 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.891115 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11683,"thread_start_us":11524,"threads_started":1}
I20260619 02:46:32.894152 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.873990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44221 (local address 127.0.0.1:57290)
0619 02:46:32.883593 (+ 9603us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:32.883612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:32.883742 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:32.884392 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:32.884400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:32.884431 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:46:32.885694 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.885706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.888287 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.888292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.890243 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:32.890256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.893369 (+ 3113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.893375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:32.893596 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.893638 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:46:32.893731 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:46:32.893859 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9483,"thread_start_us":9375,"threads_started":1}
I20260619 02:46:32.894349 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:32.874222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.7.246.66:44221)
0619 02:46:32.883331 (+ 9109us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:32.883343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:32.883861 (+ 518us) server_negotiation.cc:407] Connection header received
0619 02:46:32.883956 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:32.883963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:32.884206 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:32.885641 (+ 1435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:46:32.885883 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.888135 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.890416 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:32.892176 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:32.892339 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:32.892513 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:32.893845 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0619 02:46:32.894009 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8881,"thread_start_us":8744,"threads_started":1}
I20260619 02:46:32.895248 19898 tablet_service.cc:1467] Processing CreateTablet for tablet 397de3a702904b0daf198f10e4655374 (DEFAULT_TABLE table=test-table [id=ab1461c8e288475ca7eec50121d57191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:46:32.896371 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397de3a702904b0daf198f10e4655374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:46:32.897951 19764 tablet_service.cc:1467] Processing CreateTablet for tablet 397de3a702904b0daf198f10e4655374 (DEFAULT_TABLE table=test-table [id=ab1461c8e288475ca7eec50121d57191]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:46:32.899168 19764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397de3a702904b0daf198f10e4655374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:46:32.906492 19982 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Bootstrap starting.
I20260619 02:46:32.908311 19983 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Bootstrap starting.
I20260619 02:46:32.911567 19982 tablet_bootstrap.cc:654] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Neither blocks nor log segments found. Creating new log.
I20260619 02:46:32.912236 19983 tablet_bootstrap.cc:654] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Neither blocks nor log segments found. Creating new log.
I20260619 02:46:32.912358 19984 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Bootstrap starting.
I20260619 02:46:32.912781 19982 log.cc:824] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:46:32.913707 19983 log.cc:824] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Log is configured to *not* fsync() on all Append() calls
I20260619 02:46:32.915692 19982 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: No bootstrap required, opened a new log
I20260619 02:46:32.915719 19984 tablet_bootstrap.cc:654] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Neither blocks nor log segments found. Creating new log.
I20260619 02:46:32.915951 19982 ts_tablet_manager.cc:1397] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 02:46:32.916586 19983 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: No bootstrap required, opened a new log
I20260619 02:46:32.916822 19983 ts_tablet_manager.cc:1397] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:46:32.917104 19984 log.cc:824] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:46:32.919804 19984 tablet_bootstrap.cc:492] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: No bootstrap required, opened a new log
I20260619 02:46:32.920051 19984 ts_tablet_manager.cc:1397] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 02:46:32.926836 19984 raft_consensus.cc:348] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.927170 19984 raft_consensus.cc:374] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:46:32.927306 19984 raft_consensus.cc:729] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f78e5ec48584be0a6c481d2de80b86e, State: Initialized, Role: FOLLOWER
I20260619 02:46:32.927386 19982 raft_consensus.cc:348] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.927414 19983 raft_consensus.cc:348] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.927773 19982 raft_consensus.cc:374] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:46:32.927773 19983 raft_consensus.cc:374] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:46:32.927870 19982 raft_consensus.cc:729] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241a8a9886b34b179dd00a27c23ef6f0, State: Initialized, Role: FOLLOWER
I20260619 02:46:32.927870 19983 raft_consensus.cc:729] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f3974ca4194e4a8b070204c2e473da, State: Initialized, Role: FOLLOWER
I20260619 02:46:32.927949 19984 consensus_queue.cc:260] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.928444 19982 consensus_queue.cc:260] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.928444 19983 consensus_queue.cc:260] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:32.929210 19829 heartbeater.cc:503] Master 127.7.246.126:41785 was elected leader, sending a full tablet report...
I20260619 02:46:32.929700 19984 ts_tablet_manager.cc:1428] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260619 02:46:32.931097 19963 heartbeater.cc:503] Master 127.7.246.126:41785 was elected leader, sending a full tablet report...
I20260619 02:46:32.931310 19983 ts_tablet_manager.cc:1428] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 02:46:32.933748 19982 ts_tablet_manager.cc:1428] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
W20260619 02:46:32.946244 19830 tablet.cc:2369] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:46:33.028774 19990 raft_consensus.cc:482] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:46:33.029027 19990 raft_consensus.cc:504] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:33.030627 19990 leader_election.cc:288] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f3974ca4194e4a8b070204c2e473da (127.7.246.65:40905), 2f78e5ec48584be0a6c481d2de80b86e (127.7.246.66:44221)
I20260619 02:46:33.036659 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.030925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:40905 (local address 127.7.246.67:58165)
0619 02:46:33.031102 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:33.031123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:33.031267 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:33.031709 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:33.031714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:33.031742 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:46:33.031915 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:33.031922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.033090 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.033094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:33.034808 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:33.034824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.036171 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.036176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:33.036298 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.036334 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:46:33.036402 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:46:33.036481 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:46:33.036852 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.031053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:58165 (local address 127.7.246.65:40905)
0619 02:46:33.031289 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.031296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.031343 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:46:33.031429 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.031435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.031553 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.031700 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:46:33.032049 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.032933 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.034995 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.036036 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.036220 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.036301 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.036453 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.036588 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:46:33.037851 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397de3a702904b0daf198f10e4655374" candidate_uuid: "241a8a9886b34b179dd00a27c23ef6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f3974ca4194e4a8b070204c2e473da" is_pre_election: true
I20260619 02:46:33.038059 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.031574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:45943 (local address 127.7.246.66:44221)
0619 02:46:33.031769 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.031775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.032312 (+ 537us) server_negotiation.cc:407] Connection header received
0619 02:46:33.032484 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.032490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.032562 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.032647 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:46:33.033156 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.033948 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.035742 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.037032 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.037193 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.037303 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.037608 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.037754 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:46:33.038125 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.031321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44221 (local address 127.7.246.67:45943)
0619 02:46:33.032209 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 02:46:33.032221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:46:33.032346 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:46:33.032667 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:46:33.032672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:46:33.032713 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:46:33.032984 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:33.032991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.034091 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.034096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:33.035607 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:46:33.035617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.037208 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.037232 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:46:33.037428 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.037451 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:46:33.037591 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:46:33.037716 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":680,"threads_started":1}
I20260619 02:46:33.038437 19641 raft_consensus.cc:2454] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 241a8a9886b34b179dd00a27c23ef6f0 in term 0.
I20260619 02:46:33.039474 19854 leader_election.cc:302] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 241a8a9886b34b179dd00a27c23ef6f0, 35f3974ca4194e4a8b070204c2e473da; no voters:
I20260619 02:46:33.039544 19784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397de3a702904b0daf198f10e4655374" candidate_uuid: "241a8a9886b34b179dd00a27c23ef6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f78e5ec48584be0a6c481d2de80b86e" is_pre_election: true
I20260619 02:46:33.039829 19990 raft_consensus.cc:2781] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:46:33.039979 19990 raft_consensus.cc:482] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:46:33.040096 19990 raft_consensus.cc:3037] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:46:33.040195 19784 raft_consensus.cc:2454] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 241a8a9886b34b179dd00a27c23ef6f0 in term 0.
I20260619 02:46:33.042541 19990 raft_consensus.cc:504] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
I20260619 02:46:33.043071 19990 leader_election.cc:288] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [CANDIDATE]: Term 1 election: Requested vote from peers 35f3974ca4194e4a8b070204c2e473da (127.7.246.65:40905), 2f78e5ec48584be0a6c481d2de80b86e (127.7.246.66:44221)
I20260619 02:46:33.043532 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397de3a702904b0daf198f10e4655374" candidate_uuid: "241a8a9886b34b179dd00a27c23ef6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f3974ca4194e4a8b070204c2e473da"
I20260619 02:46:33.043598 19784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397de3a702904b0daf198f10e4655374" candidate_uuid: "241a8a9886b34b179dd00a27c23ef6f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f78e5ec48584be0a6c481d2de80b86e"
I20260619 02:46:33.043704 19641 raft_consensus.cc:3037] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:46:33.043825 19784 raft_consensus.cc:3037] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:46:33.045825 19641 raft_consensus.cc:2454] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241a8a9886b34b179dd00a27c23ef6f0 in term 1.
I20260619 02:46:33.046237 19854 leader_election.cc:302] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 241a8a9886b34b179dd00a27c23ef6f0, 35f3974ca4194e4a8b070204c2e473da; no voters:
I20260619 02:46:33.046396 19784 raft_consensus.cc:2454] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 241a8a9886b34b179dd00a27c23ef6f0 in term 1.
I20260619 02:46:33.046510 19990 raft_consensus.cc:2781] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:46:33.046999 19990 raft_consensus.cc:686] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 1 LEADER]: Becoming Leader. State: Replica: 241a8a9886b34b179dd00a27c23ef6f0, State: Running, Role: LEADER
I20260619 02:46:33.047475 19990 consensus_queue.cc:237] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } }
W20260619 02:46:33.052345 19964 tablet.cc:2369] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:46:33.051987 19473 catalog_manager.cc:5495] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 241a8a9886b34b179dd00a27c23ef6f0 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "241a8a9886b34b179dd00a27c23ef6f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "241a8a9886b34b179dd00a27c23ef6f0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 34143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 } health_report { overall_health: UNKNOWN } } }
I20260619 02:46:33.070778 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.065338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.7.246.67:34143)
0619 02:46:33.065567 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.065573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.065709 (+ 136us) server_negotiation.cc:407] Connection header received
0619 02:46:33.065936 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.065940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.066007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.066157 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:46:33.066515 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.067264 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.068712 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.069104 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.069241 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.069730 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:46:33.070207 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:46:33.070376 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.070453 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.070525 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:46:33.077863 19641 raft_consensus.cc:1261] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 1 FOLLOWER]: Refusing update from remote peer 241a8a9886b34b179dd00a27c23ef6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:46:33.077863 19784 raft_consensus.cc:1261] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 1 FOLLOWER]: Refusing update from remote peer 241a8a9886b34b179dd00a27c23ef6f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:46:33.078783 19992 consensus_queue.cc:1035] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f78e5ec48584be0a6c481d2de80b86e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:46:33.079124 19990 consensus_queue.cc:1035] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f3974ca4194e4a8b070204c2e473da" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:46:33.089450 19995 mvcc.cc:205] Tried to move back new op lower bound from 7298405142838136832 to 7298405142726373376. Current Snapshot: MvccSnapshot[applied={T|T < 7298405142838136832}]
I20260619 02:46:33.104573 19472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:46:33.104856 19472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:46:33.108759 19472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f950289a417342bd96fd54f4baa01a45: Sending DeleteTablet for 3 replicas of tablet 397de3a702904b0daf198f10e4655374
I20260619 02:46:33.110026 19898 tablet_service.cc:1514] Processing DeleteTablet for tablet 397de3a702904b0daf198f10e4655374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:46:33 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:57182
I20260619 02:46:33.110059 19764 tablet_service.cc:1514] Processing DeleteTablet for tablet 397de3a702904b0daf198f10e4655374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:46:33 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:57290
I20260619 02:46:33.110994 19621 tablet_service.cc:1514] Processing DeleteTablet for tablet 397de3a702904b0daf198f10e4655374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:46:33 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:60308
I20260619 02:46:33.111788 20007 tablet_replica.cc:330] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: stopping tablet replica
I20260619 02:46:33.112340 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 19553
I20260619 02:46:33.112294 20007 raft_consensus.cc:2229] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:46:33.112810 20007 raft_consensus.cc:2258] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:46:33.117868 20007 ts_tablet_manager.cc:1905] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:46:33.124418 20006 tablet_replica.cc:330] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: stopping tablet replica
I20260619 02:46:33.124961 20006 raft_consensus.cc:2229] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:46:33.125527 20006 raft_consensus.cc:2258] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:46:33.126403 20005 tablet_replica.cc:330] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: stopping tablet replica
I20260619 02:46:33.128166 20006 ts_tablet_manager.cc:1905] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:46:33.128605 20005 raft_consensus.cc:2229] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:46:33.128973 20007 ts_tablet_manager.cc:1918] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:46:33.129171 20007 log.cc:1196] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/397de3a702904b0daf198f10e4655374
I20260619 02:46:33.129824 20007 ts_tablet_manager.cc:1939] T 397de3a702904b0daf198f10e4655374 P 35f3974ca4194e4a8b070204c2e473da: Deleting consensus metadata
I20260619 02:46:33.131668 19460 catalog_manager.cc:4841] TS 35f3974ca4194e4a8b070204c2e473da (127.7.246.65:40905): tablet 397de3a702904b0daf198f10e4655374 (table test-table [id=ab1461c8e288475ca7eec50121d57191]) successfully deleted
I20260619 02:46:33.132059 20005 raft_consensus.cc:2258] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:46:33.133930 20005 ts_tablet_manager.cc:1905] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:46:33.135510 20006 ts_tablet_manager.cc:1918] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:46:33.135679 20006 log.cc:1196] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/397de3a702904b0daf198f10e4655374
Jun 19 02:46:33 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:46:33.136282 20006 ts_tablet_manager.cc:1939] T 397de3a702904b0daf198f10e4655374 P 2f78e5ec48584be0a6c481d2de80b86e: Deleting consensus metadata
I20260619 02:46:33.138211 19460 catalog_manager.cc:4841] TS 2f78e5ec48584be0a6c481d2de80b86e (127.7.246.66:44221): tablet 397de3a702904b0daf198f10e4655374 (table test-table [id=ab1461c8e288475ca7eec50121d57191]) successfully deleted
I20260619 02:46:33.141505 20005 ts_tablet_manager.cc:1918] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:46:33.141672 20005 log.cc:1196] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/397de3a702904b0daf198f10e4655374
I20260619 02:46:33.142172 20005 ts_tablet_manager.cc:1939] T 397de3a702904b0daf198f10e4655374 P 241a8a9886b34b179dd00a27c23ef6f0: Deleting consensus metadata
I20260619 02:46:33.143630 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.113870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.7.246.65:40905)
0619 02:46:33.114029 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.114035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.124546 (+ 10511us) server_negotiation.cc:407] Connection header received
0619 02:46:33.124956 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.124962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.125049 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.125123 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:33.127014 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.128150 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.130178 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.130802 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.130899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.137262 (+ 6363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:33.137278 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:33.137294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:33.137332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:33.140599 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.141262 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.141276 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.141279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.141414 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.141872 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.141878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.141886 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.142217 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:33.142472 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.142993 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.143216 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:46:33.143673 19458 catalog_manager.cc:4841] TS 241a8a9886b34b179dd00a27c23ef6f0 (127.7.246.67:34143): tablet 397de3a702904b0daf198f10e4655374 (table test-table [id=ab1461c8e288475ca7eec50121d57191]) successfully deleted
I20260619 02:46:33.144364 19681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:46:33.249970 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 19553
I20260619 02:46:33.281041 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 19692
Jun 19 02:46:33 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:46:33.297612 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.282009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.7.246.66:44221)
0619 02:46:33.282198 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.282206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.282231 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:46:33.282548 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.282554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.282645 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.282735 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:33.284141 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.284857 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.286298 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.286667 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.286809 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.291245 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:33.291259 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:33.291273 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:33.291305 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:33.294634 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.295288 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.295297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.295301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.295427 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.295791 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.295798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.295801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.296133 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:33.296451 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.297062 (+ 611us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.297282 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:46:33.298264 19824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:46:33.386590 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 19692
I20260619 02:46:33.406000 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 19835
Jun 19 02:46:33 dist-test-slave-9tkz krb5kdc[18996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:46:33.421898 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:46:33.406833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.7.246.67:34143)
0619 02:46:33.407025 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:46:33.407031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:46:33.407078 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:46:33.407331 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:46:33.407343 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:46:33.407448 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:46:33.407553 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:46:33.408822 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.409830 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.411028 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:46:33.411513 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:46:33.411648 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:46:33.415996 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:46:33.416008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:46:33.416021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:46:33.416055 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:46:33.419172 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.419805 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.419820 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.419822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.419970 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:46:33.420335 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:46:33.420341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:46:33.420345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:46:33.420657 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:46:33.420951 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 02:46:33.421258 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 02:46:33.421457 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:46:33.422657 19958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:46:33.535650 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 19835
I20260619 02:46:33.550837 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 19441
I20260619 02:46:33.551981 19503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:46:33.659611 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 19441
I20260619 02:46:33.680473 8153 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:46:34.016939 8153 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:46:34.017169 8153 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:46:34.362493 8153 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:46:34.362 UTC [19026] LOG: received smart shutdown request
2026-06-19 02:46:34.366 UTC [19026] LOG: background worker "logical replication launcher" (PID 19035) exited with exit code 1
2026-06-19 02:46:34.366 UTC [19030] LOG: shutting down
2026-06-19 02:46:34.524 UTC [19026] LOG: database system is shut down
2026-06-19T02:46:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49155 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:46:34 dist-test-slave-9tkz krb5kdc[20021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:46:34 dist-test-slave-9tkz krb5kdc[20021](info): set up 2 sockets
Jun 19 02:46:34 dist-test-slave-9tkz krb5kdc[20021](info): commencing operation
krb5kdc: starting...
W20260619 02:46:35.749938 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:46:35 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:46:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangeradmin_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangeradmin_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangerlookup_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangerlookup_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
I20260619 02:46:36.062736 8153 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfqHoa6/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/postgres -l logfile start
2026-06-19 02:46:38.464 UTC [20051] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:46:38.465 UTC [20051] LOG: listening on IPv4 address "127.7.246.84", port 35411
2026-06-19 02:46:38.468 UTC [20051] LOG: listening on Unix socket "/tmp/.s.PGSQL.35411"
2026-06-19 02:46:38.488 UTC [20054] LOG: database system was shut down at 2026-06-19 02:46:37 UTC
2026-06-19 02:46:38.492 UTC [20051] LOG: database system is ready to accept connections
I20260619 02:46:39.461788 8153 mini_postgres.cc:96] Postgres bound to 35411
2026-06-19 02:46:39.482 UTC [20062] FATAL: database "slave" does not exist
127.7.246.84:35411 - accepting connections
I20260619 02:46:39.483487 8153 mini_ranger.cc:161] Starting Ranger...
I20260619 02:46:39.509014 8153 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:46:39.967932 8153 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:46:39.968101 8153 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin
2026-06-19 02:46:40,253 [I] DB FLAVOR :POSTGRES
2026-06-19 02:46:40,253 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:46:40,253 [I] Checking connection..
2026-06-19 02:46:40,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:46:40,479 [I] Checking connection passed.
2026-06-19 02:46:40,479 [I] --------- Verifying version history table ---------
2026-06-19 02:46:40,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:46:40,737 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:46:40,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:46:40,989 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:46:40,990 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:46:40,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:46:41.220 UTC [20126] WARNING: there is no transaction in progress
2026-06-19 02:46:41,226 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:46:41,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:46:41,462 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:46:41,462 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:46:41,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:46:41,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:46:41,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9tkz.c.gcp-upstream.internal','N') ;"
2026-06-19 02:46:42,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:46:42,413 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:46:42,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:46:42,662 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:46:42,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:46:42,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:46:43,108 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:46:43,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:46:44.033 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.049 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.065 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.079 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.094 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.111 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.164 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.171 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.184 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.194 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.205 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.218 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.227 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.232 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.239 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44.245 UTC [20234] WARNING: there is no transaction in progress
2026-06-19 02:46:44,863 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:46:44,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:46:45,113 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:46:45,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:46:45,351 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:46:45,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:46:45,592 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:46:45,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:46:45,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:46:46,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9tkz.c.gcp-upstream.internal';"
2026-06-19 02:46:46,276 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:46:46,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:46:46,532 [I] DB_PATCHES have already been applied
I20260619 02:46:46.538326 8153 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfqHoa6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:46:46.538409 8153 mini_ranger.cc:193] Using host: 127.7.246.84
I20260619 02:46:46.553148 8153 mini_ranger.cc:227] Ranger admin URL: http://127.7.246.84:44269
Jun 19, 2026 2:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:46:47.694329 8153 mini_ranger.cc:160] Time spent starting Ranger: real 8.211s user 0.000s sys 0.038s
Jun 19, 2026 2:47:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:47:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:47:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:47:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:47:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:47:09.044880 8153 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.7.246.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.7.246.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangerkms_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.7.246.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/rangerkms_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.7.246.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.7.246.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.7.246.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/HTTP_127.7.246.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:47:09 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837229, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:47:09.207670 8153 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:47:09.226040 8153 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:47:10.022435 8153 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:47:10.022553 8153 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms
I20260619 02:47:10.022600 8153 mini_ranger_kms.cc:227] Using postgres at 127.7.246.84:35411
2026-06-19 02:47:10,366 [I] DB FLAVOR :POSTGRES
2026-06-19 02:47:10,366 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:47:10,366 [I] Checking connection
2026-06-19 02:47:10,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:47:10,596 [I] connection success
2026-06-19 02:47:10,596 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:47:10,596 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:47:10,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:47:10,850 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:47:10,851 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:47:10,851 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:47:10,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.7.246.84:35411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:47:11,091 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:47:11.524307 8153 mini_ranger_kms.cc:310] Created kms service
I20260619 02:47:11.712958 8153 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:47:11.757215 8153 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:47:11 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837229, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:12.134518 8153 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:47:12.134853 8153 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfqHoa6/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfqHoa6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:47:12.134922 8153 mini_ranger_kms.cc:240] Using host: 127.7.246.84
I20260619 02:47:12.155117 8153 mini_ranger_kms.cc:276] Ranger KMS PID: 20415
I20260619 02:47:12.155233 8153 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.7.246.84:43335
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:47:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:47:14.440199 8153 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.233s user 0.077s sys 0.278s
I20260619 02:47:14.440454 8153 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:47:14.440618 8153 mini_ranger_kms.cc:401] 127.7.246.84:43335/kms/v1/keys
Jun 19 02:47:14 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:47:14 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
Jun 19 02:47:15 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837234, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.7.246.84@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
Jun 19 02:47:17 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:47:17.572991 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40241
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:40241
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:17.901165 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:17.901463 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:17.901587 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:17.908838 20466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:17.909019 20466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:17.909113 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:17.909193 20466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:17.909242 20466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:17.919447 20466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
--ranger_kms_url=127.7.246.84:43335/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:40241
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:17.920704 20466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:17.921979 20466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:17.934067 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:17.934118 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:17.934020 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:17.939139 20466 server_base.cc:1034] running on GCE node
I20260619 02:47:17.939929 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:17.941380 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:17.942610 20466 hybrid_clock.cc:648] HybridClock initialized: now 1781837237942556 us; error 67 us; skew 500 ppm
I20260619 02:47:17.942965 20466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:17 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:17.952116 20466 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:17.953950 20466 webserver.cc:460] Webserver started at http://127.7.246.126:46565/ using document root <none> and password file <none>
I20260619 02:47:17.954722 20466 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:17.954865 20466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:17.955186 20466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:47:17 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:18.013006 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "30b6d81172314b57983a113a763e59aa"
format_stamp: "Formatted at 2026-06-19 02:47:18 on dist-test-slave-9tkz"
server_key: "1af0c32cdbd05c9d83b655fb2a6f4f85"
server_key_iv: "45e6fe997e4c88471f2046def6a41ea8"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:18.013940 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "30b6d81172314b57983a113a763e59aa"
format_stamp: "Formatted at 2026-06-19 02:47:18 on dist-test-slave-9tkz"
server_key: "1af0c32cdbd05c9d83b655fb2a6f4f85"
server_key_iv: "45e6fe997e4c88471f2046def6a41ea8"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:18.019284 20466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:47:18.034376 20481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:18.035872 20466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:18.036054 20466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "30b6d81172314b57983a113a763e59aa"
format_stamp: "Formatted at 2026-06-19 02:47:18 on dist-test-slave-9tkz"
server_key: "1af0c32cdbd05c9d83b655fb2a6f4f85"
server_key_iv: "45e6fe997e4c88471f2046def6a41ea8"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:18.036271 20466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:18.056562 20466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:18.060781 20466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:18.061160 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:18.080380 20466 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:40241
I20260619 02:47:18.080384 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:40241 every 8 connection(s)
I20260619 02:47:18.081749 20466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:18.085455 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:18.086966 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 20466
I20260619 02:47:18.087182 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:47:18.087951 8153 external_mini_cluster.cc:1428] Setting key 30dae906f1fa76b7a99c7fd1004565af
I20260619 02:47:18.102100 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Bootstrap starting.
I20260619 02:47:18.106655 20533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:18.108026 20533 log.cc:824] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:18.111631 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: No bootstrap required, opened a new log
Jun 19 02:47:18 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:18.122527 20533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:18.123029 20533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:18.123183 20533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b6d81172314b57983a113a763e59aa, State: Initialized, Role: FOLLOWER
I20260619 02:47:18.123996 20533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:18.124315 20533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:18.124496 20533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:18.124693 20533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:18.125625 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:18.090102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.7.246.126:40241)
0619 02:47:18.105165 (+ 15063us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:18.105182 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:18.105232 (+ 50us) server_negotiation.cc:407] Connection header received
0619 02:47:18.107819 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:18.107857 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:18.108389 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:18.109011 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:18.110402 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:18.111836 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:18.113159 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:18.114054 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:18.114280 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:18.118229 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:18.118237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:18.118252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:18.118294 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:18.121307 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:18.122052 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:18.122066 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:18.122072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:18.122209 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:18.122607 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:18.122611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:18.122613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:18.123287 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:18.123618 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:18.123971 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:47:18.124379 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14818,"thread_start_us":14593,"threads_started":1}
I20260619 02:47:18.128113 20533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:18.128684 20533 leader_election.cc:302] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b6d81172314b57983a113a763e59aa; no voters:
I20260619 02:47:18.129285 20533 leader_election.cc:288] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:47:18.129387 20538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:18.131363 20533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:18.136431 20538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 LEADER]: Becoming Leader. State: Replica: 30b6d81172314b57983a113a763e59aa, State: Running, Role: LEADER
I20260619 02:47:18.136929 20538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:18.147125 20540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b6d81172314b57983a113a763e59aa. Latest consensus state: current_term: 1 leader_uuid: "30b6d81172314b57983a113a763e59aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } } }
I20260619 02:47:18.147564 20540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:18.148131 20539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b6d81172314b57983a113a763e59aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } } }
I20260619 02:47:18.148468 20539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:18.170804 20547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:18.176699 20547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:18.177481 20533 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:47:18.194839 20547 catalog_manager.cc:1347] Generated new cluster ID: 71c05b0c044b4892baa00064b03a8d6e
I20260619 02:47:18.195050 20547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:18.212225 20547 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:47:18.214282 20547 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:18.222503 20547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Generated new TSK 0
I20260619 02:47:18.223675 20547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:47:18.943924 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:19 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:47:19.354264 20579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:19.354574 20579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:19.354758 20579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:19.366142 20579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:19.366300 20579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:19.366493 20579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:19.379451 20579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:19.381060 20579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:19.382823 20579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:19.400290 20589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:19.400947 20590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:19.411280 20592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:19.411716 20579 server_base.cc:1034] running on GCE node
I20260619 02:47:19.412547 20579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:19.414034 20579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:19.415390 20579 hybrid_clock.cc:648] HybridClock initialized: now 1781837239415261 us; error 112 us; skew 500 ppm
I20260619 02:47:19.415794 20579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:19 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:19.420120 20579 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:19.422037 20579 webserver.cc:460] Webserver started at http://127.7.246.65:35337/ using document root <none> and password file <none>
I20260619 02:47:19.422963 20579 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:19.423115 20579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:19.423547 20579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:47:19 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:19.442044 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "fd638fedf3304bd9a2e29525462bbc6f"
format_stamp: "Formatted at 2026-06-19 02:47:19 on dist-test-slave-9tkz"
server_key: "e67a47aac788c6a5dabe05fc8e9c0859"
server_key_iv: "56e6e56f3f49bce6c6eb748ee342a6f5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:19.443190 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "fd638fedf3304bd9a2e29525462bbc6f"
format_stamp: "Formatted at 2026-06-19 02:47:19 on dist-test-slave-9tkz"
server_key: "e67a47aac788c6a5dabe05fc8e9c0859"
server_key_iv: "56e6e56f3f49bce6c6eb748ee342a6f5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:19.451395 20579 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 02:47:19.478276 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:19.480497 20579 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.003s
I20260619 02:47:19.480763 20579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "fd638fedf3304bd9a2e29525462bbc6f"
format_stamp: "Formatted at 2026-06-19 02:47:19 on dist-test-slave-9tkz"
server_key: "e67a47aac788c6a5dabe05fc8e9c0859"
server_key_iv: "56e6e56f3f49bce6c6eb748ee342a6f5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:19.481137 20579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:19.518188 20579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:19.523638 20579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:19.524077 20579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:19.525343 20579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:19.527267 20579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:19.527400 20579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:19.527581 20579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:19.527690 20579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:19.601446 20579 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44345
I20260619 02:47:19.601469 20711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44345 every 8 connection(s)
I20260619 02:47:19.602880 20579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:19.608494 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 20579
I20260619 02:47:19.608726 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:47:19.609666 8153 external_mini_cluster.cc:1428] Setting key cc506d80eda2ec8ff0942fd6a4b62273
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Jun 19 02:47:19 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
I20260619 02:47:19.674667 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:19.610712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:39907 (local address 127.7.246.126:40241)
0619 02:47:19.633107 (+ 22395us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:19.633114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:19.640422 (+ 7308us) server_negotiation.cc:407] Connection header received
0619 02:47:19.643634 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:19.643642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:19.643769 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:19.643892 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:19.646848 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:19.648173 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:19.651029 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:19.651784 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:19.651913 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:19.663255 (+ 11342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:19.663264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:19.663270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:19.663340 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:19.667252 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:19.670260 (+ 3008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:19.670266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:19.670270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:19.670443 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:19.672340 (+ 1897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:19.672346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:19.672350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:19.672738 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:19.672895 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:19.674011 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0619 02:47:19.674250 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22226,"thread_start_us":22391,"threads_started":1}
I20260619 02:47:19.677477 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:19.610935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.65:39907)
0619 02:47:19.640010 (+ 29075us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:19.640097 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:19.643183 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:19.644613 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:19.644631 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:19.645492 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:19.646588 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:19.646619 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:19.648434 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:19.648440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:19.650831 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:19.650856 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:19.652353 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:19.652362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:19.652712 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:19.653684 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:19.653721 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:19.659049 (+ 5328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:19.667996 (+ 8947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:19.668007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:19.668028 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:19.668723 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:19.671909 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:19.671914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:19.671918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:19.672192 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:19.673015 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:19.673023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:19.673485 (+ 462us) client_negotiation.cc:769] Sending connection context
0619 02:47:19.675437 (+ 1952us) client_negotiation.cc:240] Negotiation successful
0619 02:47:19.676020 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28667,"thread_start_us":28414,"threads_started":1}
I20260619 02:47:19.679960 20712 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
I20260619 02:47:19.680318 20712 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:19.681218 20712 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:19.684165 20498 ts_manager.cc:194] Registered new tserver with Master: fd638fedf3304bd9a2e29525462bbc6f (127.7.246.65:44345)
I20260619 02:47:19.686432 20498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:39907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:47:19.697633 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:19 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
W20260619 02:47:20.141371 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:20.141783 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:20.141942 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:20.153348 20718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:20.153613 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:20.153857 20718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:20.169502 20718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:20.171476 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:20.173238 20718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:20.218681 20725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:20.220199 20726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:20.224630 20729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:20.224885 20718 server_base.cc:1034] running on GCE node
I20260619 02:47:20.225872 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:20.234745 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:20.236137 20718 hybrid_clock.cc:648] HybridClock initialized: now 1781837240236028 us; error 112 us; skew 500 ppm
I20260619 02:47:20.236603 20718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:20.240952 20718 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:20.261256 20718 webserver.cc:460] Webserver started at http://127.7.246.66:44111/ using document root <none> and password file <none>
I20260619 02:47:20.262440 20718 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:20.262669 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:20.263110 20718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:20.288084 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "916ecb7b5d8947f894017ccde4902f53"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "e4d2efbf7469281a0e07ba6eb1646728"
server_key_iv: "d5bf5462cf3e56d0e674aaaff8efd2d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.289265 20718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "916ecb7b5d8947f894017ccde4902f53"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "e4d2efbf7469281a0e07ba6eb1646728"
server_key_iv: "d5bf5462cf3e56d0e674aaaff8efd2d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.302852 20718 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260619 02:47:20.360013 20735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:20.366899 20718 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.017s sys 0.001s
I20260619 02:47:20.367220 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "916ecb7b5d8947f894017ccde4902f53"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "e4d2efbf7469281a0e07ba6eb1646728"
server_key_iv: "d5bf5462cf3e56d0e674aaaff8efd2d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.367560 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:20.415386 20718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:20.420605 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:20.420915 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:20.422035 20718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:20.423760 20718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:20.423864 20718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:20.423987 20718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:20.424043 20718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:20.465101 20718 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:41585
I20260619 02:47:20.465166 20855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:41585 every 8 connection(s)
I20260619 02:47:20.466758 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:20.472842 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 20718
I20260619 02:47:20.473075 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:47:20.473853 8153 external_mini_cluster.cc:1428] Setting key cef8c5955e430230242d90449b4e4d02
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:47:20.503391 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:20.469632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:56873 (local address 127.7.246.126:40241)
0619 02:47:20.486084 (+ 16452us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:20.486092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:20.486686 (+ 594us) server_negotiation.cc:407] Connection header received
0619 02:47:20.488896 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:20.488902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:20.488990 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:20.489118 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:20.490683 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:20.491444 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:20.493007 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:20.493415 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:20.493547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:20.496954 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:20.496958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:20.496962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:20.496995 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:20.499852 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:20.500662 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:20.500667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:20.500673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:20.500796 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:20.501229 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:20.501236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:20.501244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:20.501527 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:20.501726 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:20.502560 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 02:47:20.502900 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16294,"thread_start_us":16154,"threads_started":1}
I20260619 02:47:20.504309 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:20.470419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.66:56873)
0619 02:47:20.486489 (+ 16070us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:20.486551 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:20.488590 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:20.489391 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:20.489401 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:20.489915 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:20.490512 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:20.490531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:20.491601 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:20.491611 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:20.492859 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:20.492880 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:20.493659 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:20.493668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:20.493904 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:20.494611 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:20.494633 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:20.496720 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:20.499993 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:20.500002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:20.500020 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:20.500542 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:20.500918 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:20.500923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:20.500927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:20.501112 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:20.501704 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:20.501712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:20.502101 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 02:47:20.502401 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 02:47:20.502941 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15749,"thread_start_us":15529,"threads_started":1}
I20260619 02:47:20.506021 20856 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
I20260619 02:47:20.506366 20856 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:20.507227 20856 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:20.508941 20498 ts_manager.cc:194] Registered new tserver with Master: 916ecb7b5d8947f894017ccde4902f53 (127.7.246.66:41585)
I20260619 02:47:20.510057 20498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:56873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:47:20.539114 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:20.690618 20712 heartbeater.cc:503] Master 127.7.246.126:40241 was elected leader, sending a full tablet report...
W20260619 02:47:20.903955 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:20.904306 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:20.904480 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:20.911798 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:20.911954 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:20.912129 20862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:47:20.922362 20862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:20.923705 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:20.925081 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:20.938476 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:20.938562 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:20.938524 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:20.939543 20862 server_base.cc:1034] running on GCE node
I20260619 02:47:20.940148 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:20.941092 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:20.942301 20862 hybrid_clock.cc:648] HybridClock initialized: now 1781837240942228 us; error 70 us; skew 500 ppm
I20260619 02:47:20.942619 20862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:20.945782 20862 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:47:20.947245 20862 webserver.cc:460] Webserver started at http://127.7.246.67:41245/ using document root <none> and password file <none>
I20260619 02:47:20.948005 20862 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:20.948146 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:20.948439 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:47:20 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:20.965149 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "a219fcd4c5264413b77ab6526b1d50cc"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "4405dd4543f53bfe4136857c7b515983"
server_key_iv: "4815df08c0d5ecf4409263b7ccb44086"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.966043 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "a219fcd4c5264413b77ab6526b1d50cc"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "4405dd4543f53bfe4136857c7b515983"
server_key_iv: "4815df08c0d5ecf4409263b7ccb44086"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.971398 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:47:20.988267 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:20.989925 20862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:47:20.990074 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "a219fcd4c5264413b77ab6526b1d50cc"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "4405dd4543f53bfe4136857c7b515983"
server_key_iv: "4815df08c0d5ecf4409263b7ccb44086"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:20.990264 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:21.018110 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:21.022405 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:21.022650 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:21.023523 20862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:21.024850 20862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:21.024916 20862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:21.025020 20862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:21.025061 20862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:21.057279 20862 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:43045
I20260619 02:47:21.057297 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:43045 every 8 connection(s)
I20260619 02:47:21.058606 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:47:21.065163 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 20862
I20260619 02:47:21.065346 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:47:21.065927 8153 external_mini_cluster.cc:1428] Setting key 6e2ff76f69df11d46b1caf56517b73a9
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837240, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:21.078275 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:21.061036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:44247 (local address 127.7.246.126:40241)
0619 02:47:21.061490 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:21.061497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:21.062564 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 02:47:21.064252 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:21.064256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:21.064329 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:21.064423 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:21.065923 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.066806 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.068555 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.069147 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.069242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:21.072549 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:21.072553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:21.072556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:21.072605 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:21.075689 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.076304 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.076307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.076309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.076471 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.076839 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.076843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.076845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.077090 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:21.077212 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:21.077844 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 02:47:21.078008 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":199,"threads_started":1}
I20260619 02:47:21.079172 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:21.061663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.67:44247)
0619 02:47:21.062399 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:21.062441 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:21.064017 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:21.064621 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:21.064637 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:21.065035 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:21.065756 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:21.065775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.066957 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.066961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:21.068414 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:21.068425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.069344 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.069349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:21.069623 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:21.070325 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:21.070347 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:21.072338 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:21.075802 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:21.075808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:21.075820 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:21.076203 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:21.076566 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:21.076569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:21.076571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:21.076744 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:21.077188 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:21.077193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:21.077471 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 02:47:21.077750 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 02:47:21.078113 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":304,"threads_started":1}
I20260619 02:47:21.080777 20990 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
I20260619 02:47:21.081105 20990 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:21.081887 20990 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:21.083268 20498 ts_manager.cc:194] Registered new tserver with Master: a219fcd4c5264413b77ab6526b1d50cc (127.7.246.67:43045)
I20260619 02:47:21.084089 20498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:44247
I20260619 02:47:21.093593 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:47:21.095333 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 20579
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:47:21.111336 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:21.096241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.7.246.65:44345)
0619 02:47:21.096796 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:21.096803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:21.096849 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:47:21.096929 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:21.096935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:21.097130 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:21.097252 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:21.098626 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.099670 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.101126 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.102149 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.102277 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:21.106357 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:21.106367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:21.106380 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:21.106414 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:21.108951 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.109612 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.109620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.109623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.109809 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.110124 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.110127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.110128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.110418 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:21.110652 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:21.110901 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:47:21.111079 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260619 02:47:21.112623 20707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:21.195027 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 20579
I20260619 02:47:21.210847 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 20718
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:47:21.226810 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:21.211690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.7.246.66:41585)
0619 02:47:21.212313 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:21.212320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:21.212343 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:47:21.212407 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:21.212414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:21.212651 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:21.212775 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:21.213985 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.215072 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.216454 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.216963 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.217055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:21.221399 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:21.221410 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:21.221424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:21.221455 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:21.224133 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.224867 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.224874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.224876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.225005 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.225314 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.225317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.225318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.225746 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:21.225957 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:21.226209 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 02:47:21.226463 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":332,"threads_started":1}
I20260619 02:47:21.228009 20851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:21.307369 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 20718
I20260619 02:47:21.323200 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 20862
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:47:21.340164 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:21.324131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.7.246.67:43045)
0619 02:47:21.324730 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:21.324736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:21.324763 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:47:21.324816 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:21.324822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:21.325031 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:21.325205 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:21.326580 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.327573 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.328879 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:21.329516 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:21.329679 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:21.334979 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:21.334989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:21.335003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:21.335037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:21.337442 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.338117 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.338134 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.338136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.338260 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:21.338609 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:21.338616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:21.338619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:21.338938 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:21.339261 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:21.339568 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 02:47:21.339827 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":320,"threads_started":1}
I20260619 02:47:21.341398 20985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:21.422856 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 20862
I20260619 02:47:21.438254 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 20466
I20260619 02:47:21.439524 20528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:21.543219 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 20466
I20260619 02:47:21.565444 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40241
--webserver_interface=127.7.246.126
--webserver_port=46565
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:40241
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:21.887547 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:21.887857 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:21.887996 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:21.895190 21002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:21.895295 21002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:21.895428 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:21.895510 21002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:21.895578 21002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:21.905453 21002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:40241
--ranger_config_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:21.906761 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:21.908083 21002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:21.919958 21008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:21.920020 21007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:21.919999 21010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:21.921202 21002 server_base.cc:1034] running on GCE node
I20260619 02:47:21.922096 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:21.923506 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:21.924749 21002 hybrid_clock.cc:648] HybridClock initialized: now 1781837241924709 us; error 47 us; skew 500 ppm
I20260619 02:47:21.925196 21002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837241, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:21.928268 21002 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:21.929611 21002 webserver.cc:460] Webserver started at http://127.7.246.126:46565/ using document root <none> and password file <none>
I20260619 02:47:21.930198 21002 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:21.930330 21002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:47:21 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837241, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:21.984702 21002 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:47:21.987800 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:21.989254 21002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:21.989447 21002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "30b6d81172314b57983a113a763e59aa"
format_stamp: "Formatted at 2026-06-19 02:47:18 on dist-test-slave-9tkz"
server_key: "1af0c32cdbd05c9d83b655fb2a6f4f85"
server_key_iv: "45e6fe997e4c88471f2046def6a41ea8"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:21.990453 21002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:22.012496 21002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:22.016121 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:22.016396 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:22.034533 21002 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:40241
I20260619 02:47:22.034554 21069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:40241 every 8 connection(s)
I20260619 02:47:22.035914 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:22.037603 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21002
I20260619 02:47:22.038456 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:44345
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:40241
--webserver_port=35337
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:22.042606 21070 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:47:22.045751 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Bootstrap starting.
I20260619 02:47:22.064399 21070 log.cc:824] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:22.073143 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:47:22.073740 21070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Bootstrap complete.
I20260619 02:47:22.082475 21070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:22.083323 21070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b6d81172314b57983a113a763e59aa, State: Initialized, Role: FOLLOWER
I20260619 02:47:22.083896 21070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:22.084120 21070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:22.084245 21070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:22.084412 21070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:47:22.088117 21070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:22.088593 21070 leader_election.cc:302] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b6d81172314b57983a113a763e59aa; no voters:
I20260619 02:47:22.089219 21070 leader_election.cc:288] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:47:22.089380 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:47:22.090457 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [term 2 LEADER]: Becoming Leader. State: Replica: 30b6d81172314b57983a113a763e59aa, State: Running, Role: LEADER
I20260619 02:47:22.091053 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } }
I20260619 02:47:22.091413 21070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:22.093690 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30b6d81172314b57983a113a763e59aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } } }
I20260619 02:47:22.094086 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:22.100057 21076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b6d81172314b57983a113a763e59aa. Latest consensus state: current_term: 2 leader_uuid: "30b6d81172314b57983a113a763e59aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b6d81172314b57983a113a763e59aa" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40241 } } }
I20260619 02:47:22.100884 21076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:22.109254 21081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:22.114886 21081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:22.116277 21081 catalog_manager.cc:1259] Loaded cluster ID: 71c05b0c044b4892baa00064b03a8d6e
I20260619 02:47:22.116425 21081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:22.120394 21081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:22.121860 21081 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Loaded TSK: 0
I20260619 02:47:22.123394 21081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:47:22.127008 21070 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 02:47:22.460722 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:22.461249 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:22.461467 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:22.472571 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:22.472829 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:22.473088 21072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:22.486248 21072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:44345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=35337
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:22.487950 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:22.489341 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:22.542162 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:22.543694 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:22.544164 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:22.545354 21072 server_base.cc:1034] running on GCE node
I20260619 02:47:22.546289 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:22.549136 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:22.550480 21072 hybrid_clock.cc:648] HybridClock initialized: now 1781837242550399 us; error 61 us; skew 500 ppm
I20260619 02:47:22.550923 21072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:22 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837242, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:22.555522 21072 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:22.557526 21072 webserver.cc:460] Webserver started at http://127.7.246.65:35337/ using document root <none> and password file <none>
I20260619 02:47:22.558547 21072 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:22.558712 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:47:22 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837242, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:22.601953 21072 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.000s sys 0.006s
I20260619 02:47:22.623132 21118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:22.625478 21072 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.008s sys 0.001s
I20260619 02:47:22.625782 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "fd638fedf3304bd9a2e29525462bbc6f"
format_stamp: "Formatted at 2026-06-19 02:47:19 on dist-test-slave-9tkz"
server_key: "e67a47aac788c6a5dabe05fc8e9c0859"
server_key_iv: "56e6e56f3f49bce6c6eb748ee342a6f5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:22.627456 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:22.672166 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:22.676723 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:22.677060 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:22.678236 21072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:22.681332 21072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:22.681461 21072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:22.681618 21072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:22.681702 21072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:22.742298 21072 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44345
I20260619 02:47:22.742380 21231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44345 every 8 connection(s)
I20260619 02:47:22.743774 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:22.750797 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21072
I20260619 02:47:22.751775 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:41585
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:40241
--webserver_port=44111
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:22 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837242, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:22.796502 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:22.747387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.65:32795)
0619 02:47:22.762864 (+ 15477us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:22.762922 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:22.765394 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:22.767810 (+ 2416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:22.767819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:22.768450 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:22.769218 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:22.769245 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:22.771914 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:22.771919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:22.774026 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:22.774041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:22.775673 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:22.775683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:22.775946 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:22.776807 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:22.776838 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:22.785270 (+ 8432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:22.791078 (+ 5808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:22.791089 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:22.791108 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:22.791821 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:22.792368 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:22.792388 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:22.792393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:22.792642 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:22.793842 (+ 1200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:22.793850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:22.794346 (+ 496us) client_negotiation.cc:769] Sending connection context
0619 02:47:22.794653 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 02:47:22.795162 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15170,"thread_start_us":14968,"threads_started":1}
I20260619 02:47:22.796663 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:22.747370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:32795 (local address 127.7.246.126:40241)
0619 02:47:22.762273 (+ 14903us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:22.762294 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:22.763062 (+ 768us) server_negotiation.cc:407] Connection header received
0619 02:47:22.766078 (+ 3016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:22.766098 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:22.766602 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:22.766968 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:22.769498 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:22.771699 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:22.774214 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:22.775245 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:22.775448 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:22.785644 (+ 10196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:22.785654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:22.785671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:22.785709 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:22.789429 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:22.792017 (+ 2588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:22.792027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:22.792035 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:22.792218 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:22.792866 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:22.792872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:22.792875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:22.793580 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:22.793764 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:22.794956 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0619 02:47:22.795432 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14608,"thread_start_us":14401,"threads_started":1}
I20260619 02:47:22.800858 21232 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
I20260619 02:47:22.801223 21232 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:22.802517 21232 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
W20260619 02:47:22.805843 21232 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:40241 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:47:22.806072 21232 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:22.806350 21232 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:22.807545 21232 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:22.807895 21232 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
W20260619 02:47:22.809028 21232 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:47:23.156961 21236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:23.157359 21236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:23.157562 21236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:23.167497 21236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:23.167606 21236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:23.167779 21236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:23.180444 21236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:41585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=44111
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:23.181878 21236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:23.183154 21236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:23.199505 21250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:23.200574 21251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:23.202173 21253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:23.202719 21236 server_base.cc:1034] running on GCE node
I20260619 02:47:23.203456 21236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:23.204686 21236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:23.205956 21236 hybrid_clock.cc:648] HybridClock initialized: now 1781837243205885 us; error 43 us; skew 500 ppm
I20260619 02:47:23.206372 21236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:23.216621 21236 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:23.218742 21236 webserver.cc:460] Webserver started at http://127.7.246.66:44111/ using document root <none> and password file <none>
I20260619 02:47:23.219772 21236 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:23.219947 21236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:23.243453 21236 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 02:47:23.262272 21261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:23.264361 21236 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.010s sys 0.007s
I20260619 02:47:23.264650 21236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "916ecb7b5d8947f894017ccde4902f53"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "e4d2efbf7469281a0e07ba6eb1646728"
server_key_iv: "d5bf5462cf3e56d0e674aaaff8efd2d5"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:23.266157 21236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:23.323822 21236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:23.329135 21236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:23.329448 21236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:23.330670 21236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:23.332474 21236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:23.332633 21236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 02:47:23.332780 21236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:23.332892 21236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:23.412758 21236 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:41585
I20260619 02:47:23.412807 21375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:41585 every 8 connection(s)
I20260619 02:47:23.414569 21236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:23.422441 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21236
I20260619 02:47:23.423237 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.7.246.84:43335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:43045
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:40241
--webserver_port=41245
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:23.442669 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:23.419041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44523 (local address 127.7.246.126:40241)
0619 02:47:23.419523 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:23.419530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:23.420664 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 02:47:23.422986 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:23.422992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:23.423077 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:23.423159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:23.425436 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:23.426652 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:23.429344 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:23.430077 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:23.430186 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:23.434782 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:23.434790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:23.434797 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:23.434839 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:23.438695 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:23.439616 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:23.439623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:23.439626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:23.439764 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:23.440343 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:23.440354 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:23.440357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:23.440788 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:23.440929 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:23.441935 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0619 02:47:23.442179 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260619 02:47:23.445313 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:23.419626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.66:44523)
0619 02:47:23.420519 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:23.420562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:23.422645 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:23.423681 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:23.423696 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:23.424371 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:23.425238 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:23.425266 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:23.426882 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:23.426890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:23.429157 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:23.429174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:23.430554 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:23.430562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:23.430889 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:23.431737 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:23.431773 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:23.434551 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:23.438869 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:23.438881 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:23.438901 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:23.439485 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:23.439940 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:23.439945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:23.439948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:23.440195 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:23.441037 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:23.441047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:23.441480 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 02:47:23.442899 (+ 1419us) client_negotiation.cc:240] Negotiation successful
0619 02:47:23.443623 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":319,"threads_started":1}
I20260619 02:47:23.447379 21376 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
I20260619 02:47:23.447751 21376 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:23.448689 21376 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:23.450807 21035 ts_manager.cc:194] Registered new tserver with Master: 916ecb7b5d8947f894017ccde4902f53 (127.7.246.66:41585)
I20260619 02:47:23.453019 21035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:44523
I20260619 02:47:23.809190 21232 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:23.809644 21232 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:23.810804 21035 ts_manager.cc:194] Registered new tserver with Master: fd638fedf3304bd9a2e29525462bbc6f (127.7.246.65:44345)
I20260619 02:47:23.811893 21035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:32795
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
W20260619 02:47:23.865716 21381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:23.866194 21381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:23.866391 21381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:23.877496 21381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:23.877751 21381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:23.877985 21381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:47:23.893396 21381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.7.246.84:43335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:43045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=41245
--tserver_master_addrs=127.7.246.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:23.895352 21381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:23.897078 21381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:23.934496 21386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:23.940018 21387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:23.940137 21389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:23.940891 21381 server_base.cc:1034] running on GCE node
I20260619 02:47:23.941797 21381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:23.949834 21381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:23.953293 21381 hybrid_clock.cc:648] HybridClock initialized: now 1781837243953203 us; error 74 us; skew 500 ppm
I20260619 02:47:23.953788 21381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:23 dist-test-slave-9tkz krb5kdc[20021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:23.957948 21381 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:47:24.006006 21381 webserver.cc:460] Webserver started at http://127.7.246.67:41245/ using document root <none> and password file <none>
I20260619 02:47:24.007217 21381 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:24.007468 21381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:47:24 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:24.049764 21381 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260619 02:47:24.059906 21397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:24.061918 21381 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.003s
I20260619 02:47:24.062178 21381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "a219fcd4c5264413b77ab6526b1d50cc"
format_stamp: "Formatted at 2026-06-19 02:47:20 on dist-test-slave-9tkz"
server_key: "4405dd4543f53bfe4136857c7b515983"
server_key_iv: "4815df08c0d5ecf4409263b7ccb44086"
server_key_version: "kuduclusterkey@0"
I20260619 02:47:24.063633 21381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:24.103564 21381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:24.108652 21381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:24.109055 21381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:24.116430 21381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:24.124279 21381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:24.124435 21381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:24.124586 21381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:24.124691 21381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:47:24 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for HTTP/127.7.246.84@KRBTEST.COM
I20260619 02:47:24.292641 21381 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:43045
I20260619 02:47:24.292703 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:43045 every 8 connection(s)
I20260619 02:47:24.294497 21381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:47:24.296487 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21381
I20260619 02:47:24.333066 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.299294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.7.246.126:40241)
0619 02:47:24.310017 (+ 10723us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.310025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.310057 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:47:24.310133 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.310141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.310268 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.310410 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:24.315986 (+ 5576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.317473 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.319442 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.320346 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.320467 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.325717 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:24.325725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:24.325733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:24.325791 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:24.330065 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.330848 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.330854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.330857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.331018 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.331451 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.331456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.331460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.331781 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:24.332007 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.332407 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.332650 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10576,"thread_start_us":10446,"threads_started":1}
Jun 19 02:47:24 dist-test-slave-9tkz krb5kdc[20021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837243, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:24.358042 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.312839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40241 (local address 127.7.246.67:38787)
0619 02:47:24.326232 (+ 13393us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.326284 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.328954 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.331519 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.331531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.333241 (+ 1710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:24.334231 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.334256 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.336971 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.336977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.338995 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.339010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.340367 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.340393 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.340750 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.343326 (+ 2576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:24.343356 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:24.346353 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:24.352566 (+ 6213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:24.352577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:24.352598 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:24.353308 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:24.353893 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:24.353900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:24.353905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:24.354214 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:24.355023 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:24.355033 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:24.355478 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.355821 (+ 343us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.356466 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13060,"thread_start_us":12838,"threads_started":1}
I20260619 02:47:24.359463 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.320459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:38787 (local address 127.7.246.126:40241)
0619 02:47:24.329707 (+ 9248us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.329713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.329734 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:47:24.329782 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.329787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.329877 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.329985 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:24.330680 (+ 695us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x60c00040d0c0. stack: 00007f1c6028366e 00007f1c60283430 00007f1c5f30a5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f1c73166775 00007f1c6b17d270 00007f1c602a7358 00000000009da5bb 00007f1c602c6fd7 00007f1c602c2bea 00007f1c602c7cee 00007f1c66d36e4a 00007f1c66d36377 00007f1c66d4732d
0619 02:47:24.334583 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.336606 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.339178 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.340067 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.340222 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.346649 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:24.346656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:24.346662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:24.346712 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:24.352375 (+ 5663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.353508 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.353515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.353519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.353719 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.354371 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.354377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.354382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.354761 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:24.354919 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.358618 (+ 3699us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.358895 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6246,"spinlock_wait_cycles":7814016,"thread_start_us":6280,"threads_started":1}
I20260619 02:47:24.362346 21511 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40241
I20260619 02:47:24.362705 21511 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:24.363633 21511 heartbeater.cc:511] Master 127.7.246.126:40241 requested a full tablet report, sending...
I20260619 02:47:24.365458 21035 ts_manager.cc:194] Registered new tserver with Master: a219fcd4c5264413b77ab6526b1d50cc (127.7.246.67:43045)
I20260619 02:47:24.366643 21035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:38787
I20260619 02:47:24.374583 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:47:24.405444 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.380384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.7.246.126:40241)
0619 02:47:24.380583 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.380590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.388580 (+ 7990us) server_negotiation.cc:407] Connection header received
0619 02:47:24.388868 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.388874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.388967 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.389076 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:24.391268 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.392393 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.394519 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.395129 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.395219 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.398746 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:24.398753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:24.398758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:24.398797 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:24.402113 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.403028 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.403034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.403038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.403175 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.403775 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.403780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.403783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.404171 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:24.404380 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.404938 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.405152 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:47:24.412214 21035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:47:24.423164 21035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:47:24.457228 21376 heartbeater.cc:503] Master 127.7.246.126:40241 was elected leader, sending a full tablet report...
I20260619 02:47:24.475571 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.451276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59034)
0619 02:47:24.452901 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.452921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.453029 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.455644 (+ 2615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.455656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.455697 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:24.456232 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.456247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.459491 (+ 3244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.459499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.462668 (+ 3169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.462689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.474535 (+ 11846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.474544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.474753 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.474795 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.474967 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.475186 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"thread_start_us":848,"threads_started":1}
I20260619 02:47:24.475562 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.451948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.7.246.65:44345)
0619 02:47:24.453513 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.453522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.453573 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:47:24.453682 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.453690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.453946 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.454182 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:24.457664 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.459280 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.465084 (+ 5804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.470348 (+ 5264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.470520 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.470843 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.475018 (+ 4175us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.475194 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1812,"threads_started":1}
I20260619 02:47:24.478819 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.451239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:41585 (local address 127.0.0.1:47698)
0619 02:47:24.452025 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.452064 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.452246 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.456490 (+ 4244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.456496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.456525 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:24.457032 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.457043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.465694 (+ 8651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.465703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.468549 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.468571 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.478034 (+ 9463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.478041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.478246 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.478278 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.478360 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.478495 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":1289,"threads_started":1}
I20260619 02:47:24.480033 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.452022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.7.246.66:41585)
0619 02:47:24.454954 (+ 2932us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.454966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.454998 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:47:24.455115 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.455125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.455376 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.457217 (+ 1841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:24.457450 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.464859 (+ 7409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.471026 (+ 6167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.473219 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.479229 (+ 6010us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.479490 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.479561 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.479720 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2718,"thread_start_us":2580,"threads_started":1}
I20260619 02:47:24.480937 21167 tablet_service.cc:1467] Processing CreateTablet for tablet 91c1e311c28049e8a194baa65f52e462 (DEFAULT_TABLE table=test-table [id=de64d56f61cd4191bbe3a3e937e5f3eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:24.482954 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c1e311c28049e8a194baa65f52e462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:24.484062 21311 tablet_service.cc:1467] Processing CreateTablet for tablet 91c1e311c28049e8a194baa65f52e462 (DEFAULT_TABLE table=test-table [id=de64d56f61cd4191bbe3a3e937e5f3eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:24.485481 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.453017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:43045 (local address 127.0.0.1:35696)
0619 02:47:24.476246 (+ 23229us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.476269 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.476435 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.477517 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.477523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.477553 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:24.477940 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.477958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.480097 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.480104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.482276 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.482292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.484585 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.484592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.484788 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.484824 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.484931 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.485073 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23126,"thread_start_us":22992,"threads_started":1}
I20260619 02:47:24.485929 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c1e311c28049e8a194baa65f52e462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:24.486307 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.453392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.7.246.67:43045)
0619 02:47:24.474482 (+ 21090us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.474488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.476372 (+ 1884us) server_negotiation.cc:407] Connection header received
0619 02:47:24.476648 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.476658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.476970 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.477186 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:24.478242 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.479922 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.482505 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.484038 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.484182 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.484429 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.485745 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.485929 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20896,"thread_start_us":20744,"threads_started":1}
I20260619 02:47:24.493172 21446 tablet_service.cc:1467] Processing CreateTablet for tablet 91c1e311c28049e8a194baa65f52e462 (DEFAULT_TABLE table=test-table [id=de64d56f61cd4191bbe3a3e937e5f3eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:24.494681 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c1e311c28049e8a194baa65f52e462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:24.498848 21539 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Bootstrap starting.
I20260619 02:47:24.502277 21540 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Bootstrap starting.
I20260619 02:47:24.502385 21539 tablet_bootstrap.cc:654] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:24.503758 21539 log.cc:824] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:24.506265 21540 tablet_bootstrap.cc:654] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:24.507222 21539 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: No bootstrap required, opened a new log
I20260619 02:47:24.507524 21539 ts_tablet_manager.cc:1397] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:47:24.507544 21540 log.cc:824] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:24.511094 21540 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: No bootstrap required, opened a new log
I20260619 02:47:24.511492 21540 ts_tablet_manager.cc:1397] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:47:24.514344 21543 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Bootstrap starting.
I20260619 02:47:24.515976 21539 raft_consensus.cc:348] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.516451 21539 raft_consensus.cc:374] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:24.516593 21539 raft_consensus.cc:729] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd638fedf3304bd9a2e29525462bbc6f, State: Initialized, Role: FOLLOWER
I20260619 02:47:24.517234 21539 consensus_queue.cc:260] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.519500 21543 tablet_bootstrap.cc:654] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:24.520493 21232 heartbeater.cc:503] Master 127.7.246.126:40241 was elected leader, sending a full tablet report...
I20260619 02:47:24.520955 21543 log.cc:824] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:24.521183 21539 ts_tablet_manager.cc:1428] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 02:47:24.521314 21540 raft_consensus.cc:348] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.522085 21540 raft_consensus.cc:374] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:24.522203 21540 raft_consensus.cc:729] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916ecb7b5d8947f894017ccde4902f53, State: Initialized, Role: FOLLOWER
I20260619 02:47:24.522950 21540 consensus_queue.cc:260] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.524462 21543 tablet_bootstrap.cc:492] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: No bootstrap required, opened a new log
I20260619 02:47:24.524937 21540 ts_tablet_manager.cc:1428] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 02:47:24.525974 21543 ts_tablet_manager.cc:1397] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 02:47:24.534828 21543 raft_consensus.cc:348] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.535199 21543 raft_consensus.cc:374] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:24.535326 21543 raft_consensus.cc:729] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a219fcd4c5264413b77ab6526b1d50cc, State: Initialized, Role: FOLLOWER
I20260619 02:47:24.535979 21543 consensus_queue.cc:260] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.537199 21511 heartbeater.cc:503] Master 127.7.246.126:40241 was elected leader, sending a full tablet report...
I20260619 02:47:24.537931 21543 ts_tablet_manager.cc:1428] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20260619 02:47:24.580485 21514 tablet.cc:2369] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:47:24.601485 21546 raft_consensus.cc:482] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:24.601910 21546 raft_consensus.cc:504] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.603745 21546 leader_election.cc:288] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd638fedf3304bd9a2e29525462bbc6f (127.7.246.65:44345), a219fcd4c5264413b77ab6526b1d50cc (127.7.246.67:43045)
I20260619 02:47:24.612955 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.604216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:51585 (local address 127.7.246.65:44345)
0619 02:47:24.604455 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.604462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.605288 (+ 826us) server_negotiation.cc:407] Connection header received
0619 02:47:24.605450 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.605456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.605531 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.605652 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:24.606430 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.607451 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.610505 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.611702 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.611860 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.611933 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.612548 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.612693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:47:24.612946 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.604464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:51487 (local address 127.7.246.67:43045)
0619 02:47:24.604662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.604671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.605167 (+ 496us) server_negotiation.cc:407] Connection header received
0619 02:47:24.605451 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.605456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.605531 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.605652 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:24.606432 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.607463 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.610505 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.611719 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.611858 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.611931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.612548 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.612693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:47:24.613309 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.604183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.7.246.66:51585)
0619 02:47:24.605169 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.605185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.605305 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.605676 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.605682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.605775 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:24.606254 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.606262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.607611 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.607615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.610358 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.610372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.611982 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.611990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.612231 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.612406 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.612514 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.612755 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":399,"spinlock_wait_cycles":21888,"thread_start_us":663,"threads_started":1}
I20260619 02:47:24.613336 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.604063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:43045 (local address 127.7.246.66:51487)
0619 02:47:24.605046 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.605066 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:24.605253 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:24.605676 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:24.605682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:24.605711 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:24.606246 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.606258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.607609 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.607615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.610361 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:24.610372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.611996 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.612002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:24.612225 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.612406 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 02:47:24.612514 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:47:24.612755 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":303,"thread_start_us":555,"threads_started":1}
I20260619 02:47:24.614444 21466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c1e311c28049e8a194baa65f52e462" candidate_uuid: "916ecb7b5d8947f894017ccde4902f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a219fcd4c5264413b77ab6526b1d50cc" is_pre_election: true
I20260619 02:47:24.614459 21187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c1e311c28049e8a194baa65f52e462" candidate_uuid: "916ecb7b5d8947f894017ccde4902f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd638fedf3304bd9a2e29525462bbc6f" is_pre_election: true
I20260619 02:47:24.614944 21466 raft_consensus.cc:2454] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916ecb7b5d8947f894017ccde4902f53 in term 0.
I20260619 02:47:24.614948 21187 raft_consensus.cc:2454] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916ecb7b5d8947f894017ccde4902f53 in term 0.
I20260619 02:47:24.615561 21265 leader_election.cc:302] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916ecb7b5d8947f894017ccde4902f53, a219fcd4c5264413b77ab6526b1d50cc; no voters:
I20260619 02:47:24.615940 21546 raft_consensus.cc:2781] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:47:24.616117 21546 raft_consensus.cc:482] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:47:24.616231 21546 raft_consensus.cc:3037] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:24.618485 21546 raft_consensus.cc:504] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.619073 21546 leader_election.cc:288] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [CANDIDATE]: Term 1 election: Requested vote from peers fd638fedf3304bd9a2e29525462bbc6f (127.7.246.65:44345), a219fcd4c5264413b77ab6526b1d50cc (127.7.246.67:43045)
I20260619 02:47:24.619443 21187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c1e311c28049e8a194baa65f52e462" candidate_uuid: "916ecb7b5d8947f894017ccde4902f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd638fedf3304bd9a2e29525462bbc6f"
I20260619 02:47:24.619516 21466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c1e311c28049e8a194baa65f52e462" candidate_uuid: "916ecb7b5d8947f894017ccde4902f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a219fcd4c5264413b77ab6526b1d50cc"
I20260619 02:47:24.619630 21187 raft_consensus.cc:3037] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:24.619717 21466 raft_consensus.cc:3037] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:24.621981 21187 raft_consensus.cc:2454] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916ecb7b5d8947f894017ccde4902f53 in term 1.
I20260619 02:47:24.621986 21466 raft_consensus.cc:2454] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916ecb7b5d8947f894017ccde4902f53 in term 1.
I20260619 02:47:24.622364 21266 leader_election.cc:302] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916ecb7b5d8947f894017ccde4902f53, fd638fedf3304bd9a2e29525462bbc6f; no voters:
I20260619 02:47:24.622650 21546 raft_consensus.cc:2781] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:24.623137 21546 raft_consensus.cc:686] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 1 LEADER]: Becoming Leader. State: Replica: 916ecb7b5d8947f894017ccde4902f53, State: Running, Role: LEADER
I20260619 02:47:24.623627 21546 consensus_queue.cc:237] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } }
I20260619 02:47:24.628350 21034 catalog_manager.cc:5495] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 916ecb7b5d8947f894017ccde4902f53 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "916ecb7b5d8947f894017ccde4902f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916ecb7b5d8947f894017ccde4902f53" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 41585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 } health_report { overall_health: UNKNOWN } } }
I20260619 02:47:24.651612 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.646662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.7.246.66:41585)
0619 02:47:24.646833 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.646838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.646860 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:47:24.646944 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.646947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.647019 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.647134 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:47:24.647487 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.648226 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.649589 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.649994 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.650092 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.650537 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:47:24.651045 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:47:24.651156 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.651229 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.651293 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:47:24.659708 21187 raft_consensus.cc:1261] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 1 FOLLOWER]: Refusing update from remote peer 916ecb7b5d8947f894017ccde4902f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:24.659709 21466 raft_consensus.cc:1261] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 1 FOLLOWER]: Refusing update from remote peer 916ecb7b5d8947f894017ccde4902f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:24.660689 21550 consensus_queue.cc:1035] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd638fedf3304bd9a2e29525462bbc6f" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:24.661078 21546 consensus_queue.cc:1035] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a219fcd4c5264413b77ab6526b1d50cc" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:47:24.669741 21377 tablet.cc:2369] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:47:24.672817 21553 mvcc.cc:205] Tried to move back new op lower bound from 7298405354115309568 to 7298405353982070784. Current Snapshot: MvccSnapshot[applied={T|T < 7298405354115309568}]
I20260619 02:47:24.695128 21035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:24.695413 21035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:24.699141 21035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30b6d81172314b57983a113a763e59aa: Sending DeleteTablet for 3 replicas of tablet 91c1e311c28049e8a194baa65f52e462
I20260619 02:47:24.700312 21311 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c1e311c28049e8a194baa65f52e462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:24 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47698
I20260619 02:47:24.700325 21446 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c1e311c28049e8a194baa65f52e462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:24 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:35696
I20260619 02:47:24.701057 21167 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c1e311c28049e8a194baa65f52e462 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:24 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:59034
I20260619 02:47:24.711153 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21072
I20260619 02:47:24.712451 21565 tablet_replica.cc:330] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: stopping tablet replica
I20260619 02:47:24.713382 21565 raft_consensus.cc:2229] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:24.714059 21565 raft_consensus.cc:2258] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:24.719398 21564 tablet_replica.cc:330] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: stopping tablet replica
I20260619 02:47:24.720579 21565 ts_tablet_manager.cc:1905] T 91c1e311c28049e8a194baa65f52e462 P fd638fedf3304bd9a2e29525462bbc6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:24.721233 21564 raft_consensus.cc:2229] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:47:24.721644 21563 tablet_replica.cc:330] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: stopping tablet replica
I20260619 02:47:24.722188 21563 raft_consensus.cc:2229] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:24.722658 21563 raft_consensus.cc:2258] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:24.723347 21564 raft_consensus.cc:2258] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:24.724959 21563 ts_tablet_manager.cc:1905] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:24.728765 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.712140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.7.246.65:44345)
0619 02:47:24.712271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.712279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.712321 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:47:24.712525 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.712533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.712634 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.712735 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:24.714419 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.715893 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.717740 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.718634 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.718762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.721973 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:24.722003 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:24.722018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:24.722070 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:24.725957 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.726708 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.726716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.726719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.726859 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.727218 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.727223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.727225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.727629 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:24.727900 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.728192 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.728396 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:47:24.729732 21227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:24.731997 21564 ts_tablet_manager.cc:1905] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:24.735013 21563 ts_tablet_manager.cc:1918] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:24.735221 21563 log.cc:1196] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/91c1e311c28049e8a194baa65f52e462
I20260619 02:47:24.735752 21563 ts_tablet_manager.cc:1939] T 91c1e311c28049e8a194baa65f52e462 P a219fcd4c5264413b77ab6526b1d50cc: Deleting consensus metadata
I20260619 02:47:24.737625 21020 catalog_manager.cc:4841] TS a219fcd4c5264413b77ab6526b1d50cc (127.7.246.67:43045): tablet 91c1e311c28049e8a194baa65f52e462 (table test-table [id=de64d56f61cd4191bbe3a3e937e5f3eb]) successfully deleted
I20260619 02:47:24.738895 21564 ts_tablet_manager.cc:1918] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:24.739079 21564 log.cc:1196] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/91c1e311c28049e8a194baa65f52e462
I20260619 02:47:24.739559 21564 ts_tablet_manager.cc:1939] T 91c1e311c28049e8a194baa65f52e462 P 916ecb7b5d8947f894017ccde4902f53: Deleting consensus metadata
I20260619 02:47:24.740855 21020 catalog_manager.cc:4841] TS 916ecb7b5d8947f894017ccde4902f53 (127.7.246.66:41585): tablet 91c1e311c28049e8a194baa65f52e462 (table test-table [id=de64d56f61cd4191bbe3a3e937e5f3eb]) successfully deleted
I20260619 02:47:24.865336 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21072
I20260619 02:47:24.878917 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21236
W20260619 02:47:24.878917 21021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:44345 (error 108)
I20260619 02:47:24.882149 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.881634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59050)
0619 02:47:24.881815 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.881983 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:24.882545 21021 catalog_manager.cc:4584] TS fd638fedf3304bd9a2e29525462bbc6f (127.7.246.65:44345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91c1e311c28049e8a194baa65f52e462: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
I20260619 02:47:24.899528 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.879770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.7.246.66:41585)
0619 02:47:24.880358 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:24.880366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:24.880386 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:47:24.880454 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:24.880460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:24.880548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:24.880651 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:24.882374 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.883688 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.884993 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:24.885808 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:24.885940 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:24.892441 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:24.892458 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:24.892478 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:24.892562 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:24.896746 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.897394 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.897405 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.897410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.897625 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:24.897948 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:24.897954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:24.897958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:24.898425 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:24.898714 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:24.898878 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 02:47:24.899148 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456}
I20260619 02:47:24.900260 21371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:24.936419 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.935926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59060)
0619 02:47:24.936141 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.936246 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:47:24.937398 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.937027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59068)
0619 02:47:24.937192 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.937271 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:47:24.997983 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.997428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59076)
0619 02:47:24.997681 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.997807 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:47:24.999006 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:24.998651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59086)
0619 02:47:24.998796 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:24.998876 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:47:25.010694 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21236
I20260619 02:47:25.029129 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21381
I20260619 02:47:25.042733 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:25.029879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.7.246.67:43045)
0619 02:47:25.030057 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:25.030062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:25.030079 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:47:25.030207 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:25.030211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:25.030272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:25.030354 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:25.031595 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:25.032285 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:25.033472 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:25.033929 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:25.034034 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:25.036892 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:25.036906 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:25.036921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:25.036957 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:25.039781 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:25.040562 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:25.040569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:25.040572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:25.040693 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:25.041072 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:25.041075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:25.041078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:25.041416 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:25.041779 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:25.042267 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 02:47:25.042439 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:47:25.043393 21506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:25.080132 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:25.079555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59088)
0619 02:47:25.079855 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:25.079947 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260619 02:47:25.081202 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:25.080840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59094)
0619 02:47:25.081014 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:25.081091 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 02:47:25.130527 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21381
I20260619 02:47:25.146916 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21002
I20260619 02:47:25.148070 21065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:25.259384 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:25.258614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44345 (local address 127.0.0.1:59110)
0619 02:47:25.258961 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:25.259118 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 02:47:25.259505 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21002
I20260619 02:47:25.277755 8153 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:47:25.623049 8153 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:47:25.623224 8153 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:47:25.967804 8153 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:47:25.968 UTC [20051] LOG: received smart shutdown request
2026-06-19 02:47:25.970 UTC [20051] LOG: background worker "logical replication launcher" (PID 20060) exited with exit code 1
2026-06-19 02:47:25.971 UTC [20055] LOG: shutting down
2026-06-19 02:47:26.122 UTC [20051] LOG: database system is shut down
2026-06-19T02:47:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51600 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:47:26 dist-test-slave-9tkz krb5kdc[21579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:47:26 dist-test-slave-9tkz krb5kdc[21579](info): set up 2 sockets
Jun 19 02:47:26 dist-test-slave-9tkz krb5kdc[21579](info): commencing operation
krb5kdc: starting...
W20260619 02:47:27.350186 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:47:27 dist-test-slave-9tkz krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:47:27.566541 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41795
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:27.879762 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:27.880093 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:27.880249 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:27.887452 21592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:27.887596 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:27.887709 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:27.887823 21592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:27.887924 21592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:27.897380 21592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:27.898730 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:27.900180 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:27.911826 21597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:27.911974 21600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:27.911854 21598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:27.912453 21592 server_base.cc:1034] running on GCE node
I20260619 02:47:27.913246 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:27.914585 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:27.915804 21592 hybrid_clock.cc:648] HybridClock initialized: now 1781837247915772 us; error 47 us; skew 500 ppm
I20260619 02:47:27.916126 21592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:27 dist-test-slave-9tkz krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:27.919178 21592 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:27.920563 21592 webserver.cc:460] Webserver started at http://127.7.246.126:39533/ using document root <none> and password file <none>
I20260619 02:47:27.921216 21592 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:27.921355 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:27.921708 21592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:27.924703 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "e97d3765ded942af8fb4bda804f6c552"
format_stamp: "Formatted at 2026-06-19 02:47:27 on dist-test-slave-9tkz"
server_key: "839afad9c207f3e535b47f9a29328743"
server_key_iv: "1451df38c9b471d80dcb2e92f8bbafeb"
server_key_version: "encryptionkey@0"
I20260619 02:47:27.925629 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "e97d3765ded942af8fb4bda804f6c552"
format_stamp: "Formatted at 2026-06-19 02:47:27 on dist-test-slave-9tkz"
server_key: "839afad9c207f3e535b47f9a29328743"
server_key_iv: "1451df38c9b471d80dcb2e92f8bbafeb"
server_key_version: "encryptionkey@0"
I20260619 02:47:27.930433 21592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:47:27.933606 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:27.934876 21592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:47:27.935050 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "e97d3765ded942af8fb4bda804f6c552"
format_stamp: "Formatted at 2026-06-19 02:47:27 on dist-test-slave-9tkz"
server_key: "839afad9c207f3e535b47f9a29328743"
server_key_iv: "1451df38c9b471d80dcb2e92f8bbafeb"
server_key_version: "encryptionkey@0"
I20260619 02:47:27.935240 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:27.957563 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:27.961692 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:27.961966 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:27.980482 21592 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41795
I20260619 02:47:27.980526 21658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41795 every 8 connection(s)
I20260619 02:47:27.981791 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:27.985318 21659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:27.986426 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21592
I20260619 02:47:27.986637 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:47:27.987257 8153 external_mini_cluster.cc:1428] Setting key a9b0d0f3e82dd9cf1f9e55b00318ad69
I20260619 02:47:27.992619 21659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.7.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.125@KRBTEST.COM" created.
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:28.016920 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:27.996071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41795 (local address 127.0.0.1:54008)
0619 02:47:27.997190 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:27.997227 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:27.999043 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:27.999835 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:27.999841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.000561 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.000878 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.000894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.002524 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.002530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.003578 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.003590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.004502 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.004509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.004782 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.005508 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.005532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.009489 (+ 3957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.012731 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.012743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.012755 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.013155 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.013672 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.013678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.013682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.013842 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.014580 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.014587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.014790 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.015029 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.015594 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":221,"spinlock_wait_cycles":25984,"thread_start_us":712,"threads_started":1}
I20260619 02:47:28.016959 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:27.996070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.7.246.126:41795)
0619 02:47:27.996935 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:27.996952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:27.997344 (+ 392us) server_negotiation.cc:407] Connection header received
0619 02:47:27.999400 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:27.999408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:27.999614 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:27.999824 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.001073 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.002374 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.003719 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.004258 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.004459 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.009763 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.009771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.009781 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.009812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.012577 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.013339 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.013348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.013351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.013518 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.013962 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.013968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.013971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.014436 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.014649 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.015133 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.015486 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":599,"thread_start_us":382,"threads_started":1}
I20260619 02:47:28.020062 21659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:47:28.021046 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.020687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35440)
0619 02:47:28.020835 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.020940 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:47:28.021271 21608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
I20260619 02:47:28.022838 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.022513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35454)
0619 02:47:28.022659 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.022719 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:28.023061 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:42959: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
I20260619 02:47:28.034299 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:42959
--webserver_interface=127.7.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:28.072413 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } attempt: 1
I20260619 02:47:28.073709 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.073101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35458)
0619 02:47:28.073337 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.073499 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 02:47:28.074800 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.074367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35462)
0619 02:47:28.074552 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.074647 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 02:47:28.075023 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:42959: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
I20260619 02:47:28.143286 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } attempt: 2
I20260619 02:47:28.144527 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.143985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35468)
0619 02:47:28.144241 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.144366 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:47:28.145604 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.145186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35472)
0619 02:47:28.145339 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.145417 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:28.145785 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:42959: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
I20260619 02:47:28.237078 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } attempt: 3
I20260619 02:47:28.238507 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.237869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35478)
0619 02:47:28.238181 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.238316 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 02:47:28.239720 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.239364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35482)
0619 02:47:28.239519 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.239596 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 02:47:28.239933 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:42959: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
W20260619 02:47:28.368219 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:28.368527 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:28.368700 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:28.376066 21666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:28.376189 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:28.376291 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:28.376374 21666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:28.376436 21666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:28.383210 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } attempt: 4
I20260619 02:47:28.384697 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.384150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35484)
0619 02:47:28.384431 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.384552 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 02:47:28.385735 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.385331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:35494)
0619 02:47:28.385486 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.385600 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:28.385908 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:42959: Network error: Client connection negotiation failed: client connection to 127.7.246.125:42959: connect: Connection refused (error 111)
I20260619 02:47:28.387197 21666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:28.388589 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:28.390110 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:28.403102 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:28.403203 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:28.403256 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:28.403895 21666 server_base.cc:1034] running on GCE node
I20260619 02:47:28.404556 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:28.405964 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:28.407166 21666 hybrid_clock.cc:648] HybridClock initialized: now 1781837248407110 us; error 52 us; skew 500 ppm
I20260619 02:47:28.407495 21666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:28.413707 21666 init.cc:377] Logged in from keytab as kudu/127.7.246.125@KRBTEST.COM (short username kudu)
I20260619 02:47:28.415019 21666 webserver.cc:460] Webserver started at http://127.7.246.125:35309/ using document root <none> and password file <none>
I20260619 02:47:28.415604 21666 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:28.415695 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:28.415927 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:28.418749 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data/instance:
uuid: "b70b68e22484462fab75eb6e573121ea"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "3e52731045a87ffa4bdb65b7be2ead1b"
server_key_iv: "811b980720bbdce04b16ce19d76f05a6"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.419457 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal/instance:
uuid: "b70b68e22484462fab75eb6e573121ea"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "3e52731045a87ffa4bdb65b7be2ead1b"
server_key_iv: "811b980720bbdce04b16ce19d76f05a6"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.424185 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 02:47:28.427333 21681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:28.428507 21666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:47:28.428673 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "b70b68e22484462fab75eb6e573121ea"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "3e52731045a87ffa4bdb65b7be2ead1b"
server_key_iv: "811b980720bbdce04b16ce19d76f05a6"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.428841 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:28.443995 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:28.448076 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:28.448285 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:28.466074 21666 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.125:42959
I20260619 02:47:28.466096 21732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.125:42959 every 8 connection(s)
I20260619 02:47:28.467387 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
I20260619 02:47:28.470690 21733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:28.475020 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21666
I20260619 02:47:28.475206 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-1/wal/instance
I20260619 02:47:28.475669 8153 external_mini_cluster.cc:1428] Setting key 1478593a6f8255d061f14f9d94048731
I20260619 02:47:28.476931 21733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.124@KRBTEST.COM; defaulting to no policy
I20260619 02:47:28.498687 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.479416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.7.246.126:41795)
0619 02:47:28.479592 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.479597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.480749 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 02:47:28.482768 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.482772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.482837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.482918 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.484843 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.485464 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.487805 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.488196 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.488289 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.492314 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.492318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.492322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.492357 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.495752 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.496465 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.496471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.496485 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.496630 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.497009 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.497014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.497020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.497402 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.497584 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.498265 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.498425 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:47:28.499722 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.479856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41795 (local address 127.0.0.1:52650)
0619 02:47:28.480600 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.480638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.482481 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.483195 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.483210 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.483950 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.484683 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.484706 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.485642 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.485650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.487650 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.487672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.488408 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.488417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.488761 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.489453 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.489485 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.492092 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.495884 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.495890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.495902 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.496326 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.496735 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.496738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.496740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.496884 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.497517 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.497523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.497908 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.498139 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.498474 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.124@KRBTEST.COM" created.
I20260619 02:47:28.501536 21733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:47:28.516638 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.502223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:56520)
0619 02:47:28.503106 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.503124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.503247 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.503721 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.503727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.504224 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.504419 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.504431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.506174 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.506182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.507572 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.507583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.508327 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.508332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.508545 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.509121 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.509136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.511286 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.514287 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.514294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.514298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.514642 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.515026 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.515031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.515034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.515174 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.515718 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.515724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.515856 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.515992 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.516259 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769}
I20260619 02:47:28.516770 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.502521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.7.246.125:42959)
0619 02:47:28.503225 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.503234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.503268 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:47:28.503328 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.503338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.503569 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.503716 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.504578 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.506018 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.507698 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.508195 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.508321 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.511478 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.511487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.511499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.511525 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.514167 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.514758 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.514772 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.514775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.514874 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.515271 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.515278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.515281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.515603 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.515811 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.516128 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.516369 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":475,"thread_start_us":333,"threads_started":1}
I20260619 02:47:28.519279 21733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:47:28.520337 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.519929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39458)
0619 02:47:28.520086 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.520193 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 02:47:28.520565 21683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.522094 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.521736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39462)
0619 02:47:28.521921 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.521992 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 02:47:28.522303 21733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
I20260619 02:47:28.528098 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:36339
--webserver_interface=127.7.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:33453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:28.571710 21733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 1
I20260619 02:47:28.572957 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.572470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39470)
0619 02:47:28.572712 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.572812 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:47:28.574044 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.573729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39486)
0619 02:47:28.573876 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.573942 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:28.574285 21733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.642501 21733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 2
I20260619 02:47:28.643553 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.643122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39494)
0619 02:47:28.643311 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.643405 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:47:28.644613 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.644260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39506)
0619 02:47:28.644418 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.644479 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:47:28.644800 21733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.685204 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } attempt: 5
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:47:28.698740 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.686071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.7.246.125:42959)
0619 02:47:28.686219 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.686223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.686241 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:47:28.686461 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.686465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.686523 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.686615 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.687569 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.688243 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.689917 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.690317 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.690405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.693792 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.693797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.693800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.693828 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.696266 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.696867 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.696878 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.696881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.696990 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.697354 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.697360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.697363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.697667 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.697806 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.698213 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.698413 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:47:28.698758 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.685915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:56530)
0619 02:47:28.686149 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.686174 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.686317 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.686698 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.686704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.687197 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.687429 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.687441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.688370 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.688376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.689761 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.689775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.690461 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.690467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.690646 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.691245 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.691267 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.693600 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.696395 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.696401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.696406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.696751 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.697104 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.697108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.697112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.697242 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.697772 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.697776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.697924 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.698082 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.698382 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 02:47:28.699738 21659 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:47:28.700872 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.700496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39518)
0619 02:47:28.700680 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.700755 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:47:28.701874 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.701507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39528)
0619 02:47:28.701686 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.701757 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 02:47:28.702132 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.735996 21733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 3
I20260619 02:47:28.737054 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.736610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39536)
0619 02:47:28.736822 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.736910 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 02:47:28.738181 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.737762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39544)
0619 02:47:28.737999 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.738077 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260619 02:47:28.738497 21733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.753358 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 1
I20260619 02:47:28.754487 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.753932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39548)
0619 02:47:28.754159 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.754280 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 02:47:28.755541 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.755102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39564)
0619 02:47:28.755286 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.755393 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 02:47:28.755765 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.824012 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 2
I20260619 02:47:28.825265 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.824698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39566)
0619 02:47:28.824950 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.825063 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:47:28.826355 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.825915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39580)
0619 02:47:28.826090 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.826200 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:47:28.826584 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
W20260619 02:47:28.853634 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:28.853911 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:28.854094 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:28.861356 21740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:28.861457 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:28.861545 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:28.861675 21740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:28.861738 21740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:28.871241 21740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:33453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41795,127.7.246.125:42959,127.7.246.124:36339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:28.872481 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:28.873780 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:47:28.881716 21733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 4
I20260619 02:47:28.882863 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.882315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39590)
0619 02:47:28.882554 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.882669 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:47:28.884011 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.883643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39598)
0619 02:47:28.883799 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.883879 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 02:47:28.884271 21733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
W20260619 02:47:28.885353 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:28.885319 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:28.885702 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:28.887079 21740 server_base.cc:1034] running on GCE node
I20260619 02:47:28.887923 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:28.889313 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:28.890529 21740 hybrid_clock.cc:648] HybridClock initialized: now 1781837248890483 us; error 44 us; skew 500 ppm
I20260619 02:47:28.890868 21740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:28.893834 21740 init.cc:377] Logged in from keytab as kudu/127.7.246.124@KRBTEST.COM (short username kudu)
I20260619 02:47:28.895172 21740 webserver.cc:460] Webserver started at http://127.7.246.124:44089/ using document root <none> and password file <none>
I20260619 02:47:28.895836 21740 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:28.895975 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:28.896275 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:28.899078 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data/instance:
uuid: "1d10d63976d141e58d9ccd6e7e09bf90"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "166264d25f564c0ddde283bfd499bdd3"
server_key_iv: "92132f6ec7fada3821ac8e998440d0b0"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.899866 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal/instance:
uuid: "1d10d63976d141e58d9ccd6e7e09bf90"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "166264d25f564c0ddde283bfd499bdd3"
server_key_iv: "92132f6ec7fada3821ac8e998440d0b0"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.905089 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 02:47:28.908390 21755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:28.909722 21740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:28.909883 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "1d10d63976d141e58d9ccd6e7e09bf90"
format_stamp: "Formatted at 2026-06-19 02:47:28 on dist-test-slave-9tkz"
server_key: "166264d25f564c0ddde283bfd499bdd3"
server_key_iv: "92132f6ec7fada3821ac8e998440d0b0"
server_key_version: "encryptionkey@0"
I20260619 02:47:28.910058 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:28.922272 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:28.926060 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:28.926334 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:28.932801 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 3
I20260619 02:47:28.934047 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.933495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39600)
0619 02:47:28.933814 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.933900 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260619 02:47:28.935235 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.934800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39610)
0619 02:47:28.935030 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.935123 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 02:47:28.935520 21659 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:36339: Network error: Client connection negotiation failed: client connection to 127.7.246.124:36339: connect: Connection refused (error 111)
I20260619 02:47:28.944802 21740 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.124:36339
I20260619 02:47:28.944834 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.124:36339 every 8 connection(s)
I20260619 02:47:28.946277 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
I20260619 02:47:28.948490 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21740
I20260619 02:47:28.948668 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837047186258-8153-0/minicluster-data/master-2/wal/instance
I20260619 02:47:28.949096 8153 external_mini_cluster.cc:1428] Setting key 3c484ef8757c6627f7c8a995feb397f9
I20260619 02:47:28.951088 21807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:28.958132 21807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:28.971110 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.950829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.7.246.124:36339)
0619 02:47:28.951609 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.951620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.951658 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:47:28.953222 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.953242 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.953714 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.954199 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.955693 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.956979 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.958451 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.959178 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.959373 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.963863 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.963870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.963879 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.963915 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.967077 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.967776 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.967787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.967793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.967939 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.968320 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.968326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.968329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.968892 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.969204 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.969650 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.970170 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":368,"threads_started":1}
I20260619 02:47:28.974577 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.960577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.7.246.126:41795)
0619 02:47:28.960786 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.960794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.961304 (+ 510us) server_negotiation.cc:407] Connection header received
0619 02:47:28.961500 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.961512 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.961637 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.961748 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.963109 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.963794 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.965444 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.965985 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.966124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.969194 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.969198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.969201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.969234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.971947 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.972636 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.972643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.972647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.972785 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.973172 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.973180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.973183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.973424 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.973613 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.974093 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.974300 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:47:28.974962 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.960537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41795 (local address 127.0.0.1:52652)
0619 02:47:28.961175 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.961208 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.961371 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.961804 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.961810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.962511 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.962879 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.962894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.963968 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.963972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.965286 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.965298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.966197 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.966210 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.966383 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.966976 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.966995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.969002 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.972082 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.972093 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.972099 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.972506 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.972908 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.972912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.972916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.973061 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.973589 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.973598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.973766 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.973978 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.974391 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":298,"threads_started":1}
I20260619 02:47:28.976646 21807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:28.991961 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.977730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.7.246.125:42959)
0619 02:47:28.977935 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.977941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.977957 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:47:28.978014 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.978017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.978077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.978208 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.978839 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.979481 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.981141 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.981835 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.981921 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.985529 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.985535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.985540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.985613 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.989026 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.989852 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.989857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.989861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.990001 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.990448 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.990453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.990457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.990780 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.990893 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.991367 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.991590 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:47:28.992661 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.977369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:42959 (local address 127.0.0.1:56540)
0619 02:47:28.977606 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.977631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.977722 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.978230 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.978234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.978574 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.978692 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.978699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.979619 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.979625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.980963 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.980978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.982139 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.982145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.982329 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.982882 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.982902 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:28.985329 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:28.989168 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.989172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.989176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.989643 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.990141 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:28.990144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:28.990148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:28.990310 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:28.990980 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:28.990985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:28.991119 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 02:47:28.992214 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0619 02:47:28.992442 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 02:47:28.993609 21807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:47:28.996217 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.973782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.7.246.126:41795)
0619 02:47:28.974863 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.974868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.974882 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:47:28.974929 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.974932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.974979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.975336 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.976805 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.977825 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.980384 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.981338 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.981495 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.989014 (+ 7519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:28.989022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:28.989029 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:28.989079 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:28.993018 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.993840 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.993846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.993849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.993994 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:28.994407 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:28.994414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:28.994417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:28.994782 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:28.995048 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:28.995363 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 02:47:28.995960 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":18112,"threads_started":1}
Jun 19 02:47:28 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
Jun 19 02:47:29 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:47:29.007290 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.994492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39626)
0619 02:47:28.994762 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:28.994779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:28.994887 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:28.995178 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:28.995181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:28.995634 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:28.995804 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.995813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.996864 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.996869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.998221 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:28.998236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.999039 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.999047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:28.999264 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:28.999788 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:28.999809 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:29.001946 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:29.004932 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.004940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.004943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.005302 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.005691 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.005696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.005700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.005849 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.006375 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:29.006380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:29.006535 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 02:47:29.006714 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 02:47:29.007003 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260619 02:47:29.007644 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.994500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.7.246.124:36339)
0619 02:47:28.994662 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.994667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.994913 (+ 246us) server_negotiation.cc:407] Connection header received
0619 02:47:28.994970 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.994975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.995043 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.995153 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.995948 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.996744 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.998346 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:28.998871 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:28.998967 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.002114 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.002127 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.002130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.002155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.004803 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.005421 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.005424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.005426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.005535 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.005995 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.005999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.006001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.006253 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.006356 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.006788 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.007218 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":58}
I20260619 02:47:29.012348 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:28.997919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.7.246.125:42959)
0619 02:47:28.998093 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:28.998098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:28.998124 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:47:28.998260 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:28.998263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:28.998344 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:28.998493 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:28.999924 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.000677 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.002068 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.002496 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.002627 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.006938 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.006942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.006945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.006972 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.010013 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.010758 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.010761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.010764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.010881 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.011251 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.011254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.011256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.011477 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.011668 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.011956 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.012112 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:47:29.013675 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: Bootstrap starting.
I20260619 02:47:29.016849 21807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:29.016868 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:47:29.017932 21807 log.cc:824] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:29.020577 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: No bootstrap required, opened a new log
I20260619 02:47:29.033686 21807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.034118 21807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:29.034237 21807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d10d63976d141e58d9ccd6e7e09bf90, State: Initialized, Role: FOLLOWER
I20260619 02:47:29.034968 21807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.046495 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.021754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.7.246.126:41795)
0619 02:47:29.021925 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:29.021933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:29.022981 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 02:47:29.023258 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:29.023266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:29.023366 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:29.023451 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:29.026391 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.027807 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.030542 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.031358 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.031466 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.037341 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.037350 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.037357 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.037424 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.042188 (+ 4764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.043499 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.043508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.043515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.043703 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.044254 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.044260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.044264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.044655 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.044901 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.045600 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.045816 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:47:29.047114 21807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:29.049698 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.050148 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.051059 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.023059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.7.246.125:42959)
0619 02:47:29.023212 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:29.023217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:29.032721 (+ 9504us) server_negotiation.cc:407] Connection header received
0619 02:47:29.032882 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:29.032886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:29.032953 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:29.033049 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:29.035313 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.036446 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.038546 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.039185 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.039284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.043287 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.043293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.043297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.043334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.047090 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.048041 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.048047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.048052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.048216 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.048840 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.048848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.048854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.049320 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.049657 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.050489 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.050720 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:47:29.051311 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.022223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.7.246.124:36339)
0619 02:47:29.022456 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:29.022465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:29.032319 (+ 9854us) server_negotiation.cc:407] Connection header received
0619 02:47:29.033329 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:29.033335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:29.033409 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:29.033474 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:29.035708 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.036721 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.038725 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.039409 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.039497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.043325 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.043332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.043336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.043370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.047740 (+ 4370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.048920 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.048925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.048928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.049098 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.049585 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.049595 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.049598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.050003 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.050242 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.050760 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.050965 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 02:47:29.062230 21832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:47:29.062378 21832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:47:29.059719 21740 master_runner.cc:186] Error getting master registration for 127.7.246.126:41795: OK, instance_id { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" instance_seqno: 1781837247975338 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:47:29.070631 21740 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:47:29.112802 21659 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 4
Jun 19 02:47:29 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837247, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:47:29.127431 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.113414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39654)
0619 02:47:29.113637 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:29.113653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:29.113776 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:29.114191 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:29.114195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:29.114640 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:29.114795 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:29.114803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.115908 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.115913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:29.116856 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:29.116867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.117626 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.117632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:29.117802 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.118451 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:29.118468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:29.121121 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:29.125136 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.125140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.125143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.125490 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.125951 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.125954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.125957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.126090 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.126712 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:29.126715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:29.126852 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 02:47:29.126974 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 02:47:29.127171 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:47:29.127830 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.113683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.7.246.124:36339)
0619 02:47:29.113884 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:29.113889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:29.113905 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:47:29.113957 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:29.113960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:29.114049 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:29.114163 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:29.114932 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.115778 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.116998 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.117463 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.117606 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.121312 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.121320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.121325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.121386 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.124992 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.125641 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.125647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.125652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.125835 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.126201 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.126207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.126211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.126598 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.126761 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.127123 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.127372 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:47:29.133900 21659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: Bootstrap starting.
I20260619 02:47:29.138235 21659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:29.139588 21659 log.cc:824] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:29.142976 21659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: No bootstrap required, opened a new log
I20260619 02:47:29.154274 21659 raft_consensus.cc:348] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.154677 21659 raft_consensus.cc:374] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:29.154800 21659 raft_consensus.cc:729] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97d3765ded942af8fb4bda804f6c552, State: Initialized, Role: FOLLOWER
I20260619 02:47:29.155449 21659 consensus_queue.cc:260] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.156316 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.156629 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.157778 21659 sys_catalog.cc:564] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:47:29.166791 21846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:47:29.170776 21846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:47:29.178438 21761 tablet.cc:2369] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:47:29.166766 21592 master_runner.cc:186] Error getting master registration for 127.7.246.125:42959: OK, instance_id { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" instance_seqno: 1781837248460927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:47:29.178781 21592 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:47:29.183576 21733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } attempt: 5
Jun 19 02:47:29 dist-test-slave-9tkz krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837248, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:47:29.200889 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.184219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:36339 (local address 127.0.0.1:39662)
0619 02:47:29.184504 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:29.184523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:29.184645 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:29.185026 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:29.185031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:29.185538 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:29.185817 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:29.185829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.187304 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.187310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:29.188735 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:29.188750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.190010 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.190016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:29.190244 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.190982 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:29.191006 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:29.193819 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:29.198060 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.198065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.198069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.198623 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.199030 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:29.199034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:29.199038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:29.199201 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:29.199831 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:29.199836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:29.200004 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 02:47:29.200165 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 02:47:29.200491 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 02:47:29.201277 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:29.184296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.7.246.124:36339)
0619 02:47:29.184538 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:29.184544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:29.184586 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:47:29.184783 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:29.184789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:29.184879 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:29.184978 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:29.185967 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.187126 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.188914 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:29.189660 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:29.189783 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:29.194029 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:29.194040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:29.194045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:29.194089 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:29.197919 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.198748 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.198754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.198759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.198909 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:29.199318 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:29.199323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:29.199326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:29.199697 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:29.199915 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:29.200422 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 02:47:29.200662 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 02:47:29.207334 21733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: Bootstrap starting.
I20260619 02:47:29.211651 21733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:29.212949 21733 log.cc:824] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: Log is configured to *not* fsync() on all Append() calls
W20260619 02:47:29.214771 21613 tablet.cc:2369] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:47:29.216199 21733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: No bootstrap required, opened a new log
I20260619 02:47:29.227080 21733 raft_consensus.cc:348] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.227473 21733 raft_consensus.cc:374] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:29.227587 21733 raft_consensus.cc:729] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70b68e22484462fab75eb6e573121ea, State: Initialized, Role: FOLLOWER
I20260619 02:47:29.228231 21733 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.229183 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.229661 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.230485 21733 sys_catalog.cc:564] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:29.233675 21835 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:29.233971 21835 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.235392 21835 leader_election.cc:288] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b70b68e22484462fab75eb6e573121ea (127.7.246.125:42959), 1d10d63976d141e58d9ccd6e7e09bf90 (127.7.246.124:36339)
I20260619 02:47:29.236109 21708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e97d3765ded942af8fb4bda804f6c552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70b68e22484462fab75eb6e573121ea" is_pre_election: true
I20260619 02:47:29.236371 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e97d3765ded942af8fb4bda804f6c552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" is_pre_election: true
I20260619 02:47:29.236670 21708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97d3765ded942af8fb4bda804f6c552 in term 0.
I20260619 02:47:29.236948 21782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97d3765ded942af8fb4bda804f6c552 in term 0.
I20260619 02:47:29.237479 21608 leader_election.cc:302] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70b68e22484462fab75eb6e573121ea, e97d3765ded942af8fb4bda804f6c552; no voters:
I20260619 02:47:29.237859 21835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:47:29.237999 21835 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:47:29.238125 21835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:29.240649 21835 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.241710 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e97d3765ded942af8fb4bda804f6c552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d10d63976d141e58d9ccd6e7e09bf90"
I20260619 02:47:29.241750 21708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e97d3765ded942af8fb4bda804f6c552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70b68e22484462fab75eb6e573121ea"
I20260619 02:47:29.241923 21708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:29.241923 21782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:29.245234 21782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97d3765ded942af8fb4bda804f6c552 in term 1.
I20260619 02:47:29.245235 21708 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97d3765ded942af8fb4bda804f6c552 in term 1.
I20260619 02:47:29.245807 21609 leader_election.cc:302] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d10d63976d141e58d9ccd6e7e09bf90, e97d3765ded942af8fb4bda804f6c552; no voters:
I20260619 02:47:29.247973 21835 leader_election.cc:288] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [CANDIDATE]: Term 1 election: Requested vote from peers b70b68e22484462fab75eb6e573121ea (127.7.246.125:42959), 1d10d63976d141e58d9ccd6e7e09bf90 (127.7.246.124:36339)
I20260619 02:47:29.248168 21835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:29.248286 21835 raft_consensus.cc:686] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [term 1 LEADER]: Becoming Leader. State: Replica: e97d3765ded942af8fb4bda804f6c552, State: Running, Role: LEADER
I20260619 02:47:29.248814 21835 consensus_queue.cc:237] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } }
I20260619 02:47:29.253865 21861 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:47:29.254153 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97d3765ded942af8fb4bda804f6c552. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.254433 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:29.263633 21863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:29.266074 21863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:29.276211 21782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [term 1 FOLLOWER]: Refusing update from remote peer e97d3765ded942af8fb4bda804f6c552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:29.276371 21708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [term 1 FOLLOWER]: Refusing update from remote peer e97d3765ded942af8fb4bda804f6c552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:29.277266 21835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:29.277705 21860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:29.305209 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97d3765ded942af8fb4bda804f6c552. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.305573 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.311652 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97d3765ded942af8fb4bda804f6c552. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.311933 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.312314 21866 mvcc.cc:205] Tried to move back new op lower bound from 7298405373025771520 to 7298405372934602752. Current Snapshot: MvccSnapshot[applied={T|T < 7298405373025771520}]
I20260619 02:47:29.314096 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.314685 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.317170 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.317420 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:47:29.318293 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.318704 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:29.318975 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e97d3765ded942af8fb4bda804f6c552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97d3765ded942af8fb4bda804f6c552" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41795 } } peers { permanent_uuid: "b70b68e22484462fab75eb6e573121ea" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 42959 } } peers { permanent_uuid: "1d10d63976d141e58d9ccd6e7e09bf90" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 36339 } } }
I20260619 02:47:29.319239 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:29.320778 21863 catalog_manager.cc:1347] Generated new cluster ID: 9fcd5be7c3e7453788ebd6ff3bf22078
I20260619 02:47:29.320873 21863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:29.337424 21863 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:47:29.339587 21863 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:29.354902 21863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e97d3765ded942af8fb4bda804f6c552: Generated new TSK 0
I20260619 02:47:29.356105 21863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:47:30.064291 21832 catalog_manager.cc:1259] Loaded cluster ID: 9fcd5be7c3e7453788ebd6ff3bf22078
I20260619 02:47:30.064389 21832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: loading cluster ID for follower catalog manager: success
I20260619 02:47:30.066844 21832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: acquiring CA information for follower catalog manager: success
I20260619 02:47:30.068575 21832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d10d63976d141e58d9ccd6e7e09bf90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:47:30.258227 21861 catalog_manager.cc:1259] Loaded cluster ID: 9fcd5be7c3e7453788ebd6ff3bf22078
I20260619 02:47:30.258342 21861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: loading cluster ID for follower catalog manager: success
I20260619 02:47:30.261152 21861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: acquiring CA information for follower catalog manager: success
I20260619 02:47:30.262991 21861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b70b68e22484462fab75eb6e573121ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:47:30.433462 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21592
I20260619 02:47:30.434788 21654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:30.548092 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21592
I20260619 02:47:30.569465 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21666
I20260619 02:47:30.570765 21728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:30.687561 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21666
I20260619 02:47:30.703397 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21740
I20260619 02:47:30.704535 21802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:30.809638 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21740
2026-06-19T02:47:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4551 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:47:30 dist-test-slave-9tkz krb5kdc[21882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:47:30 dist-test-slave-9tkz krb5kdc[21882](info): set up 2 sockets
Jun 19 02:47:30 dist-test-slave-9tkz krb5kdc[21882](info): commencing operation
krb5kdc: starting...
W20260619 02:47:31.906080 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:47:32.121503 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:35633
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:35633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:32.438898 21895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:32.439165 21895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:32.439253 21895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:32.446416 21895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:32.446519 21895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:32.446563 21895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:32.446615 21895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:32.446659 21895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:32.456456 21895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:35633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:35633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:32.457681 21895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:32.458889 21895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:32.470149 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:32.470194 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:32.470448 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:32.471326 21895 server_base.cc:1034] running on GCE node
I20260619 02:47:32.472028 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:32.473357 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:32.474581 21895 hybrid_clock.cc:648] HybridClock initialized: now 1781837252474559 us; error 51 us; skew 500 ppm
I20260619 02:47:32.474905 21895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837252, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:32.477744 21895 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:32.479118 21895 webserver.cc:460] Webserver started at http://127.7.246.126:44073/ using document root <none> and password file <none>
I20260619 02:47:32.479753 21895 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:32.479887 21895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:32.480217 21895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:32.483047 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "da1bf6bed0ad439f9850ee9db158e7b7"
format_stamp: "Formatted at 2026-06-19 02:47:32 on dist-test-slave-9tkz"
server_key: "981daf013692a00dd9c4df66d7b54e9f"
server_key_iv: "cad6a765c83d584f40e72683259fee63"
server_key_version: "encryptionkey@0"
I20260619 02:47:32.483794 21895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "da1bf6bed0ad439f9850ee9db158e7b7"
format_stamp: "Formatted at 2026-06-19 02:47:32 on dist-test-slave-9tkz"
server_key: "981daf013692a00dd9c4df66d7b54e9f"
server_key_iv: "cad6a765c83d584f40e72683259fee63"
server_key_version: "encryptionkey@0"
I20260619 02:47:32.488559 21895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:47:32.491961 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:32.493259 21895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:32.493439 21895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "da1bf6bed0ad439f9850ee9db158e7b7"
format_stamp: "Formatted at 2026-06-19 02:47:32 on dist-test-slave-9tkz"
server_key: "981daf013692a00dd9c4df66d7b54e9f"
server_key_iv: "cad6a765c83d584f40e72683259fee63"
server_key_version: "encryptionkey@0"
I20260619 02:47:32.493701 21895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:32.510072 21895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:32.513875 21895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:32.514146 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:32.532299 21895 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:35633
I20260619 02:47:32.532330 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:35633 every 8 connection(s)
I20260619 02:47:32.533679 21895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:32.537269 21962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:32.541225 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 21895
I20260619 02:47:32.541435 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:47:32.542122 8153 external_mini_cluster.cc:1428] Setting key b237852b1cb88a27f3eef54cfd9f64b5
I20260619 02:47:32.547915 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7: Bootstrap starting.
I20260619 02:47:32.551388 21962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:32.552364 21962 log.cc:824] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:32.555312 21962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7: No bootstrap required, opened a new log
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:32.564893 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:32.543906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.7.246.126:35633)
0619 02:47:32.544669 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:32.544680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:32.544718 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:47:32.546654 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:32.546676 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:32.547020 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:32.547399 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:32.549047 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:32.550948 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:32.552370 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:32.553175 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:32.553378 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:32.557954 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:32.557961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:32.557974 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:32.558017 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:32.561005 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:32.561882 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:32.561888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:32.561893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:32.562057 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:32.562428 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:32.562431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:32.562433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:32.562921 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:32.563146 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:32.563586 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 02:47:32.563913 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":338,"threads_started":1}
I20260619 02:47:32.564817 21962 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } }
I20260619 02:47:32.565285 21962 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:32.565421 21962 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1bf6bed0ad439f9850ee9db158e7b7, State: Initialized, Role: FOLLOWER
I20260619 02:47:32.566082 21962 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } }
I20260619 02:47:32.566361 21962 raft_consensus.cc:388] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:32.566534 21962 raft_consensus.cc:482] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:32.566735 21962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:32.568982 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:47:32.569489 21962 raft_consensus.cc:504] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } }
I20260619 02:47:32.570111 21962 leader_election.cc:302] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da1bf6bed0ad439f9850ee9db158e7b7; no voters:
I20260619 02:47:32.570729 21962 leader_election.cc:288] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:47:32.570830 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:32.572192 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [term 1 LEADER]: Becoming Leader. State: Replica: da1bf6bed0ad439f9850ee9db158e7b7, State: Running, Role: LEADER
I20260619 02:47:32.572732 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } }
I20260619 02:47:32.572980 21962 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:32.575412 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da1bf6bed0ad439f9850ee9db158e7b7. Latest consensus state: current_term: 1 leader_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } } }
I20260619 02:47:32.575634 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bf6bed0ad439f9850ee9db158e7b7" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 35633 } } }
I20260619 02:47:32.575879 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:32.575899 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:32.576792 21976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:32.582959 21976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:32.594058 21976 catalog_manager.cc:1347] Generated new cluster ID: 3dde1fe0003e45c283eb34c355d06753
I20260619 02:47:32.594172 21976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:32.617187 21976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:47:32.618537 21976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:32.624305 21976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da1bf6bed0ad439f9850ee9db158e7b7: Generated new TSK 0
I20260619 02:47:32.625173 21976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:47:32.681779 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 21895
I20260619 02:47:32.682910 21957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:32.786520 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 21895
2026-06-19T02:47:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1970 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21990](info): set up 2 sockets
Jun 19 02:47:32 dist-test-slave-9tkz krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20260619 02:47:33.878166 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:47:34 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:47:34.096854 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:34.411312 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:34.411818 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:34.411954 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:34.419421 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:34.419543 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:34.419588 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:34.419637 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:34.419687 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:34.429879 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:34.431147 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:34.432391 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:34.444042 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:34.444087 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:34.444120 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:34.444934 22003 server_base.cc:1034] running on GCE node
I20260619 02:47:34.445848 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:34.447396 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:34.448668 22003 hybrid_clock.cc:648] HybridClock initialized: now 1781837254448593 us; error 63 us; skew 500 ppm
I20260619 02:47:34.449047 22003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:34 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:34.455775 22003 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:34.457343 22003 webserver.cc:460] Webserver started at http://127.7.246.126:43793/ using document root <none> and password file <none>
I20260619 02:47:34.458237 22003 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:34.458389 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:34.458711 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:34.462028 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "481dc8b3c8e34339b7fdb682a647ae6d"
format_stamp: "Formatted at 2026-06-19 02:47:34 on dist-test-slave-9tkz"
server_key: "b0664ff9e4ce47fbbe2e7647b507448c"
server_key_iv: "28d8394e4215c87bed28c83610fdb3fe"
server_key_version: "encryptionkey@0"
I20260619 02:47:34.462788 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "481dc8b3c8e34339b7fdb682a647ae6d"
format_stamp: "Formatted at 2026-06-19 02:47:34 on dist-test-slave-9tkz"
server_key: "b0664ff9e4ce47fbbe2e7647b507448c"
server_key_iv: "28d8394e4215c87bed28c83610fdb3fe"
server_key_version: "encryptionkey@0"
I20260619 02:47:34.467849 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:47:34.471186 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:34.472453 22003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:47:34.472636 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "481dc8b3c8e34339b7fdb682a647ae6d"
format_stamp: "Formatted at 2026-06-19 02:47:34 on dist-test-slave-9tkz"
server_key: "b0664ff9e4ce47fbbe2e7647b507448c"
server_key_iv: "28d8394e4215c87bed28c83610fdb3fe"
server_key_version: "encryptionkey@0"
I20260619 02:47:34.472836 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:34.491030 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:34.494944 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:34.495223 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:34.513242 22003 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34285
I20260619 02:47:34.513276 22069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34285 every 8 connection(s)
I20260619 02:47:34.514644 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:34.517184 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22003
I20260619 02:47:34.517403 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:47:34.518144 8153 external_mini_cluster.cc:1428] Setting key 9a4c65d3cee46dd194045c6d9f2d6ea6
I20260619 02:47:34.518641 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:34.532902 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap starting.
Jun 19 02:47:34 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:34.536051 22070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:34.537098 22070 log.cc:824] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:34.539971 22070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: No bootstrap required, opened a new log
I20260619 02:47:34.542908 22073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:34.520109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.7.246.126:34285)
0619 02:47:34.520955 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:34.520966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:34.521007 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:47:34.523198 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:34.523221 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:34.523672 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:34.524046 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:34.525854 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:34.527135 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:34.529006 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:34.529856 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:34.530020 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:34.534947 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:34.534954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:34.534969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:34.535004 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:34.538728 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:34.539460 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:34.539466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:34.539473 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:34.539636 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:34.540046 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:34.540051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:34.540055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:34.540639 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:34.540972 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:34.541412 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 02:47:34.541890 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":414,"threads_started":1}
I20260619 02:47:34.548234 22070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:34.548599 22070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:34.548703 22070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Initialized, Role: FOLLOWER
I20260619 02:47:34.549255 22070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:34.549523 22070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:34.549767 22070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:34.549952 22070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:34.552104 22070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:34.552580 22070 leader_election.cc:302] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481dc8b3c8e34339b7fdb682a647ae6d; no voters:
I20260619 02:47:34.553201 22070 leader_election.cc:288] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:47:34.553313 22075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:34.554636 22075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 LEADER]: Becoming Leader. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Running, Role: LEADER
I20260619 02:47:34.555164 22075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:34.555316 22070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:34.557853 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481dc8b3c8e34339b7fdb682a647ae6d. Latest consensus state: current_term: 1 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:34.558208 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:34.558734 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:34.559068 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:34.560706 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:34.565420 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:34.574753 22082 catalog_manager.cc:1347] Generated new cluster ID: 364eecc567bc45faa0e3be28c1fbecda
I20260619 02:47:34.574879 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:34.590019 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:47:34.591462 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:34.609242 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Generated new TSK 0
I20260619 02:47:34.610201 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:47:34.672791 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:35.009047 22097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:35.009398 22097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:35.009603 22097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:35.017688 22097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:35.017833 22097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:35.018035 22097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:35.027772 22097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:35.029065 22097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:35.030470 22097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:35.045243 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:35.045610 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:35.045264 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:35.046274 22097 server_base.cc:1034] running on GCE node
I20260619 02:47:35.046904 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:35.047906 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:35.049125 22097 hybrid_clock.cc:648] HybridClock initialized: now 1781837255049073 us; error 41 us; skew 500 ppm
I20260619 02:47:35.049420 22097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:35 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837255, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:35.052636 22097 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:35.054183 22097 webserver.cc:460] Webserver started at http://127.7.246.65:43433/ using document root <none> and password file <none>
I20260619 02:47:35.054836 22097 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:35.054951 22097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:35.055277 22097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:35.058585 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "d21db929613e48019e782049d140a84a"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "70b375622f5ff42e44fba9108fffed59"
server_key_iv: "2be44f0c110f83b1c5f589072a9f2f49"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.059405 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "d21db929613e48019e782049d140a84a"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "70b375622f5ff42e44fba9108fffed59"
server_key_iv: "2be44f0c110f83b1c5f589072a9f2f49"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.065122 22097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 02:47:35.068657 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.070086 22097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 02:47:35.070262 22097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "d21db929613e48019e782049d140a84a"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "70b375622f5ff42e44fba9108fffed59"
server_key_iv: "2be44f0c110f83b1c5f589072a9f2f49"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.070477 22097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:35.089874 22097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:35.093742 22097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:35.093982 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:35.094851 22097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:35.096124 22097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:35.096194 22097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.096277 22097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:35.096321 22097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.129151 22097 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33007
I20260619 02:47:35.129195 22224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33007 every 8 connection(s)
I20260619 02:47:35.130508 22097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:35.138859 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22097
I20260619 02:47:35.139081 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:47:35.139726 8153 external_mini_cluster.cc:1428] Setting key 5a995f480575de046ed1833aa5d5c773
Jun 19 02:47:35 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837255, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:35.151772 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:35.132992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:60759 (local address 127.7.246.126:34285)
0619 02:47:35.133438 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:35.133444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:35.134445 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 02:47:35.136183 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:35.136187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:35.136250 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:35.136344 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:35.137884 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.138774 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.140492 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.140951 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.141128 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:35.144549 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:35.144560 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:35.144568 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:35.144617 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:35.148485 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:35.149218 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:35.149227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:35.149233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:35.149395 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:35.149946 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:35.149949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:35.149958 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:35.150249 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:35.150370 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:35.151316 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 02:47:35.151505 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260619 02:47:35.153177 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:35.133534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:60759)
0619 02:47:35.134290 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:35.134328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:35.135931 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:35.136598 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:35.136606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:35.137090 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:35.137724 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:35.137745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.138934 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.138938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:35.140317 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:35.140330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.141190 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.141195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:35.141378 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:35.142171 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:35.142198 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:35.144311 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:35.148642 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:35.148648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:35.148661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:35.149075 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:35.149532 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:35.149535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:35.149538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:35.149784 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:35.150385 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:35.150396 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:35.150857 (+ 461us) client_negotiation.cc:769] Sending connection context
0619 02:47:35.151168 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 02:47:35.151775 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":254,"threads_started":1}
I20260619 02:47:35.154807 22225 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:35.155114 22225 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:35.155875 22225 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:35.158013 22035 ts_manager.cc:194] Registered new tserver with Master: d21db929613e48019e782049d140a84a (127.7.246.65:33007)
I20260619 02:47:35.159631 22035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:60759
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:47:35.192775 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:35.512576 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:35.512975 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:35.513144 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:35.520404 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:35.520546 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:35.520727 22231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:35.530521 22231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:35.531822 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:35.533090 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:35.546078 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:35.546043 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:35.546555 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:35.546962 22231 server_base.cc:1034] running on GCE node
I20260619 02:47:35.547612 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:35.548527 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:35.549722 22231 hybrid_clock.cc:648] HybridClock initialized: now 1781837255549673 us; error 37 us; skew 500 ppm
I20260619 02:47:35.550097 22231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:35 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837255, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:35.553314 22231 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:35.554731 22231 webserver.cc:460] Webserver started at http://127.7.246.66:42531/ using document root <none> and password file <none>
I20260619 02:47:35.555495 22231 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:35.555619 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:35.555963 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:35.558923 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "c487c33639a44f0886a11ab6f53abd2b"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "d1b9033ca4a6d658be546bdec2dbd600"
server_key_iv: "06fdec810891f418e7f745495ad58a34"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.559890 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "c487c33639a44f0886a11ab6f53abd2b"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "d1b9033ca4a6d658be546bdec2dbd600"
server_key_iv: "06fdec810891f418e7f745495ad58a34"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.565090 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 02:47:35.568522 22246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.569876 22231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:35.570041 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "c487c33639a44f0886a11ab6f53abd2b"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "d1b9033ca4a6d658be546bdec2dbd600"
server_key_iv: "06fdec810891f418e7f745495ad58a34"
server_key_version: "encryptionkey@0"
I20260619 02:47:35.570220 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:35.586421 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:35.590237 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:35.590526 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:35.591519 22231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:35.592916 22231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:35.593003 22231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.593129 22231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:35.593200 22231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:35.626675 22231 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:37165
I20260619 02:47:35.626730 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:37165 every 8 connection(s)
I20260619 02:47:35.627915 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:35.633250 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22231
I20260619 02:47:35.633432 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:47:35.634060 8153 external_mini_cluster.cc:1428] Setting key fb9329168e8cfc72947e41f4e8f1fc2a
Jun 19 02:47:35 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837255, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:35.648986 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:35.630363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:36467 (local address 127.7.246.126:34285)
0619 02:47:35.630536 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:35.630541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:35.631679 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 02:47:35.633649 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:35.633655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:35.633745 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:35.633860 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:35.635505 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.636469 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.638060 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.638857 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.638998 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:35.642595 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:35.642603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:35.642611 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:35.642657 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:35.645743 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:35.646684 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:35.646691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:35.646693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:35.646834 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:35.647319 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:35.647322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:35.647324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:35.647572 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:35.647685 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:35.648581 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 02:47:35.648749 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:47:35.650641 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:35.630821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:36467)
0619 02:47:35.631510 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:35.631555 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:35.633298 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:35.634055 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:35.634063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:35.634717 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:35.635340 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:35.635358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.636622 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.636627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:35.637922 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:35.637934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:35.639066 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:35.639071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:35.639279 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:35.639958 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:35.639984 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:35.642368 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:35.645872 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:35.645884 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:35.645904 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:35.646546 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:35.646941 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:35.646949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:35.646955 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:35.647207 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:35.647705 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:35.647715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:35.648197 (+ 482us) client_negotiation.cc:769] Sending connection context
0619 02:47:35.648498 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 02:47:35.649080 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
I20260619 02:47:35.652145 22359 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:35.652457 22359 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:35.653198 22359 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:35.654745 22035 ts_manager.cc:194] Registered new tserver with Master: c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:35.655776 22035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:36467
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:47:35.686849 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:36.002837 22364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:36.003108 22364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:36.003265 22364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:36.010453 22364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:36.010557 22364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:36.010782 22364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:47:36.020364 22364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:36.021618 22364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:36.022893 22364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:36.035434 22369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:36.035360 22372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:36.035344 22370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:36.036338 22364 server_base.cc:1034] running on GCE node
I20260619 02:47:36.036872 22364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:36.037755 22364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:36.038969 22364 hybrid_clock.cc:648] HybridClock initialized: now 1781837256038918 us; error 35 us; skew 500 ppm
I20260619 02:47:36.039299 22364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:36 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837256, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:36.042404 22364 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:47:36.043725 22364 webserver.cc:460] Webserver started at http://127.7.246.67:46495/ using document root <none> and password file <none>
I20260619 02:47:36.044390 22364 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:36.044520 22364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:36.044814 22364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:36.047628 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "f2353d3c6a7544d891f3495d1451da5d"
format_stamp: "Formatted at 2026-06-19 02:47:36 on dist-test-slave-9tkz"
server_key: "1ba071acf59e81524c044a161330baf9"
server_key_iv: "1a80da62fe61d1228f22bbd465bffd54"
server_key_version: "encryptionkey@0"
I20260619 02:47:36.048388 22364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "f2353d3c6a7544d891f3495d1451da5d"
format_stamp: "Formatted at 2026-06-19 02:47:36 on dist-test-slave-9tkz"
server_key: "1ba071acf59e81524c044a161330baf9"
server_key_iv: "1a80da62fe61d1228f22bbd465bffd54"
server_key_version: "encryptionkey@0"
I20260619 02:47:36.053280 22364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 02:47:36.057017 22379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:36.058554 22364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:36.058698 22364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "f2353d3c6a7544d891f3495d1451da5d"
format_stamp: "Formatted at 2026-06-19 02:47:36 on dist-test-slave-9tkz"
server_key: "1ba071acf59e81524c044a161330baf9"
server_key_iv: "1a80da62fe61d1228f22bbd465bffd54"
server_key_version: "encryptionkey@0"
I20260619 02:47:36.058877 22364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:36.097087 22364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:36.101105 22364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:36.101392 22364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:36.102406 22364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:36.103780 22364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:36.103866 22364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:36.103976 22364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:36.104027 22364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:36.138854 22364 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39101
I20260619 02:47:36.138854 22491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39101 every 8 connection(s)
I20260619 02:47:36.140201 22364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:47:36.149207 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22364
I20260619 02:47:36.149390 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:47:36.149858 8153 external_mini_cluster.cc:1428] Setting key 318a5b86dfb4ab78662e603c391a90d3
Jun 19 02:47:36 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837256, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:36.160586 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.142789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:38811 (local address 127.7.246.126:34285)
0619 02:47:36.142979 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.142983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.144099 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 02:47:36.146072 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.146080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.146176 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.146312 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:36.147907 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.148703 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.150708 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.151256 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.151421 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.155139 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:36.155144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:36.155148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:36.155180 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:36.157950 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.158631 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.158635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.158637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.158763 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.159166 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.159170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.159172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.159436 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:36.159566 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.160158 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.160316 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:47:36.161650 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.143233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:38811)
0619 02:47:36.143931 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.143971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.145786 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.146550 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.146559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.147103 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:36.147713 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.147739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.148919 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.148931 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.150533 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.150553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.151510 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.151519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.151837 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.152613 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:36.152635 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:36.154948 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:36.158082 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:36.158089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:36.158108 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:36.158518 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:36.158853 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:36.158856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:36.158858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:36.159042 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:36.159547 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:36.159553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:36.159843 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.160053 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.160419 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260619 02:47:36.162523 22225 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:36.163123 22492 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:36.163491 22492 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:36.164399 22492 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:36.165750 22035 ts_manager.cc:194] Registered new tserver with Master: f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101)
I20260619 02:47:36.166488 22035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:38811
I20260619 02:47:36.177284 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:47:36.193442 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.181774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.7.246.126:34285)
0619 02:47:36.181982 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.181990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.182378 (+ 388us) server_negotiation.cc:407] Connection header received
0619 02:47:36.182516 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.182519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.182579 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.182658 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:36.183906 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.184607 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.185958 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.186347 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.186440 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.188604 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:36.188609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:36.188612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:36.188642 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:36.191163 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.191741 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.191744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.191747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.191887 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.192213 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.192217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.192219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.192559 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:36.192745 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.192982 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.193130 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:47:36.197345 22035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:47:36.204038 22035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:47:36.239137 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.215563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39101 (local address 127.0.0.1:41496)
0619 02:47:36.226399 (+ 10836us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.226470 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.226702 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.230104 (+ 3402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.230111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.230142 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:36.230610 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.230628 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.233656 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.233665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.236624 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.236639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.238038 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.238044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.238280 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.238332 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.238462 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.238658 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10491,"mutex_wait_us":3,"thread_start_us":17125,"threads_started":1}
I20260619 02:47:36.239492 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.216322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.7.246.67:39101)
0619 02:47:36.228990 (+ 12668us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.229004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.229048 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:47:36.229172 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.229183 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.229482 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.229726 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:36.231110 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.233439 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.236755 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.237816 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.237951 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.238202 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.238974 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.239131 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12410,"thread_start_us":14445,"threads_started":1}
I20260619 02:47:36.241788 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.227837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:57596)
0619 02:47:36.232993 (+ 5156us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.233014 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.233146 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.234195 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.234200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.234227 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:36.234763 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.234776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.236683 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.236688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.238669 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.238681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.240966 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.240972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.241135 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.241177 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.241268 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.241401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4993,"mutex_wait_us":40,"thread_start_us":3831,"threads_started":1}
I20260619 02:47:36.241825 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.228216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.7.246.65:33007)
0619 02:47:36.233758 (+ 5542us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.233764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.233785 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:47:36.233855 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.233860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.234029 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.234209 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:36.234976 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.236544 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.238853 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.240732 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.240860 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.241071 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.241329 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.241504 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5395,"thread_start_us":5201,"threads_started":1}
I20260619 02:47:36.243430 22427 tablet_service.cc:1467] Processing CreateTablet for tablet 015769abd79a44f68dae2e491bcf141f (DEFAULT_TABLE table=test-table [id=486d5334d5f640d1b061f9e82b7131b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:36.244199 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.216189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.7.246.66:37165)
0619 02:47:36.234837 (+ 18648us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.234847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.234876 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:47:36.234972 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.234978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.235191 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.235996 (+ 805us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:36.236152 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.237466 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.240856 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.242398 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.243448 (+ 1050us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.243624 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.243696 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.243866 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18494,"thread_start_us":18345,"threads_started":1}
I20260619 02:47:36.244835 22427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015769abd79a44f68dae2e491bcf141f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:36.244829 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.215707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37165 (local address 127.0.0.1:59698)
0619 02:47:36.231407 (+ 15700us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.231430 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.231564 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.235384 (+ 3820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.235389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.235409 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:36.235847 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.235859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.237651 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.237657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.240706 (+ 3049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.240718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.242650 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.242657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.242822 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.242867 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.242932 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.244475 (+ 1543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15540,"mutex_wait_us":1,"thread_start_us":11212,"threads_started":1}
I20260619 02:47:36.245636 22160 tablet_service.cc:1467] Processing CreateTablet for tablet 015769abd79a44f68dae2e491bcf141f (DEFAULT_TABLE table=test-table [id=486d5334d5f640d1b061f9e82b7131b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:36.246994 22160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015769abd79a44f68dae2e491bcf141f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:36.248314 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 015769abd79a44f68dae2e491bcf141f (DEFAULT_TABLE table=test-table [id=486d5334d5f640d1b061f9e82b7131b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:36.249665 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015769abd79a44f68dae2e491bcf141f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:36.255697 22510 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Bootstrap starting.
I20260619 02:47:36.258837 22510 tablet_bootstrap.cc:654] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:36.259850 22510 log.cc:824] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:36.262075 22512 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Bootstrap starting.
I20260619 02:47:36.263216 22510 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: No bootstrap required, opened a new log
I20260619 02:47:36.263482 22510 ts_tablet_manager.cc:1397] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 02:47:36.266016 22512 tablet_bootstrap.cc:654] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:36.266307 22513 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Bootstrap starting.
I20260619 02:47:36.267238 22512 log.cc:824] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:36.269693 22513 tablet_bootstrap.cc:654] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:36.270196 22512 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: No bootstrap required, opened a new log
I20260619 02:47:36.270457 22512 ts_tablet_manager.cc:1397] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 02:47:36.270816 22513 log.cc:824] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:36.273141 22510 raft_consensus.cc:348] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.273666 22510 raft_consensus.cc:374] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:36.273723 22513 tablet_bootstrap.cc:492] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: No bootstrap required, opened a new log
I20260619 02:47:36.273777 22510 raft_consensus.cc:729] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2353d3c6a7544d891f3495d1451da5d, State: Initialized, Role: FOLLOWER
I20260619 02:47:36.273967 22513 ts_tablet_manager.cc:1397] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:47:36.274386 22510 consensus_queue.cc:260] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.279413 22512 raft_consensus.cc:348] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.279794 22512 raft_consensus.cc:374] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:36.279917 22512 raft_consensus.cc:729] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21db929613e48019e782049d140a84a, State: Initialized, Role: FOLLOWER
I20260619 02:47:36.280486 22512 consensus_queue.cc:260] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.282075 22512 ts_tablet_manager.cc:1428] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260619 02:47:36.282876 22492 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:36.283267 22510 ts_tablet_manager.cc:1428] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 02:47:36.285290 22513 raft_consensus.cc:348] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.285741 22513 raft_consensus.cc:374] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:36.285859 22513 raft_consensus.cc:729] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c487c33639a44f0886a11ab6f53abd2b, State: Initialized, Role: FOLLOWER
I20260619 02:47:36.286412 22513 consensus_queue.cc:260] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.287531 22359 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:36.288069 22513 ts_tablet_manager.cc:1428] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 02:47:36.307780 22517 raft_consensus.cc:482] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:36.308065 22517 raft_consensus.cc:504] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.309737 22517 leader_election.cc:288] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:36.316048 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.310234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:50815 (local address 127.7.246.66:37165)
0619 02:47:36.310426 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.310434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.310997 (+ 563us) server_negotiation.cc:407] Connection header received
0619 02:47:36.311214 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.311219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.311285 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.311377 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:36.311924 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.312627 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.314390 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.315146 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.315325 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.315398 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.315670 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.315787 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:47:36.316121 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.310065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37165 (local address 127.7.246.65:50815)
0619 02:47:36.310888 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.310907 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.311075 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.311398 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.311402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.311435 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:36.311790 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.311800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.312746 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.312754 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.314247 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.314258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.315356 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.315361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.315504 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.315527 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.315644 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.315775 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":1017,"threads_started":1}
I20260619 02:47:36.316499 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.310182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:47277 (local address 127.7.246.67:39101)
0619 02:47:36.310368 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.310372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.311490 (+ 1118us) server_negotiation.cc:407] Connection header received
0619 02:47:36.311712 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.311724 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.311784 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.311882 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:36.312370 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.313070 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.314839 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.315592 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.315782 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.315856 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.316132 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.316258 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:47:36.316602 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.310255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39101 (local address 127.7.246.65:47277)
0619 02:47:36.311381 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:36.311395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:36.311573 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:36.311900 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:36.311904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:36.311929 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:36.312224 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.312233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.313219 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.313224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.314670 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:36.314682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.315819 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.315825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:36.315977 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.315999 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:47:36.316103 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:47:36.316228 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":379,"threads_started":1}
I20260619 02:47:36.317464 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015769abd79a44f68dae2e491bcf141f" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c487c33639a44f0886a11ab6f53abd2b" is_pre_election: true
I20260619 02:47:36.317694 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015769abd79a44f68dae2e491bcf141f" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2353d3c6a7544d891f3495d1451da5d" is_pre_election: true
I20260619 02:47:36.318080 22314 raft_consensus.cc:2454] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 0.
I20260619 02:47:36.318197 22447 raft_consensus.cc:2454] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 0.
I20260619 02:47:36.318905 22114 leader_election.cc:302] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c487c33639a44f0886a11ab6f53abd2b, d21db929613e48019e782049d140a84a, f2353d3c6a7544d891f3495d1451da5d; no voters:
I20260619 02:47:36.319319 22517 raft_consensus.cc:2781] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:47:36.319473 22517 raft_consensus.cc:482] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:47:36.319587 22517 raft_consensus.cc:3037] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:36.321772 22517 raft_consensus.cc:504] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.322362 22517 leader_election.cc:288] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 election: Requested vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:36.322783 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015769abd79a44f68dae2e491bcf141f" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2353d3c6a7544d891f3495d1451da5d"
I20260619 02:47:36.322824 22314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015769abd79a44f68dae2e491bcf141f" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c487c33639a44f0886a11ab6f53abd2b"
I20260619 02:47:36.322983 22447 raft_consensus.cc:3037] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:36.323032 22314 raft_consensus.cc:3037] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:36.325176 22447 raft_consensus.cc:2454] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 1.
I20260619 02:47:36.325224 22314 raft_consensus.cc:2454] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 1.
I20260619 02:47:36.325590 22114 leader_election.cc:302] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d21db929613e48019e782049d140a84a, f2353d3c6a7544d891f3495d1451da5d; no voters:
I20260619 02:47:36.325886 22517 raft_consensus.cc:2781] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:36.326272 22517 raft_consensus.cc:686] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 1 LEADER]: Becoming Leader. State: Replica: d21db929613e48019e782049d140a84a, State: Running, Role: LEADER
I20260619 02:47:36.326767 22517 consensus_queue.cc:237] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } }
I20260619 02:47:36.331365 22034 catalog_manager.cc:5495] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a reported cstate change: term changed from 0 to 1, leader changed from <none> to d21db929613e48019e782049d140a84a (127.7.246.65). New cstate: current_term: 1 leader_uuid: "d21db929613e48019e782049d140a84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } health_report { overall_health: HEALTHY } } }
I20260619 02:47:36.348584 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.343411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.7.246.65:33007)
0619 02:47:36.343595 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.343599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.343713 (+ 114us) server_negotiation.cc:407] Connection header received
0619 02:47:36.343872 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.343875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.343976 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.344093 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:47:36.344401 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.345108 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.346621 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.347019 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.347108 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.347539 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:47:36.348051 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:47:36.348180 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.348315 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.348376 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:47:36.356281 22314 raft_consensus.cc:1261] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Refusing update from remote peer d21db929613e48019e782049d140a84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:36.356300 22447 raft_consensus.cc:1261] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Refusing update from remote peer d21db929613e48019e782049d140a84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:36.357264 22521 consensus_queue.cc:1035] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:36.357687 22517 consensus_queue.cc:1035] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:36.369518 22524 mvcc.cc:205] Tried to move back new op lower bound from 7298405402025533440 to 7298405401918132224. Current Snapshot: MvccSnapshot[applied={T|T < 7298405402025533440}]
I20260619 02:47:36.371593 22526 mvcc.cc:205] Tried to move back new op lower bound from 7298405402025533440 to 7298405401918132224. Current Snapshot: MvccSnapshot[applied={T|T < 7298405402025533440}]
I20260619 02:47:36.372293 22525 mvcc.cc:205] Tried to move back new op lower bound from 7298405402025533440 to 7298405401918132224. Current Snapshot: MvccSnapshot[applied={T|T < 7298405402025533440}]
I20260619 02:47:36.387292 22035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:36.387594 22035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:36.391404 22035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Sending DeleteTablet for 3 replicas of tablet 015769abd79a44f68dae2e491bcf141f
I20260619 02:47:36.392488 22427 tablet_service.cc:1514] Processing DeleteTablet for tablet 015769abd79a44f68dae2e491bcf141f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:36 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:41496
I20260619 02:47:36.392997 22294 tablet_service.cc:1514] Processing DeleteTablet for tablet 015769abd79a44f68dae2e491bcf141f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:36 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:59698
I20260619 02:47:36.393409 22160 tablet_service.cc:1514] Processing DeleteTablet for tablet 015769abd79a44f68dae2e491bcf141f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:36 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:57596
I20260619 02:47:36.394358 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22097
I20260619 02:47:36.408064 22534 tablet_replica.cc:330] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: stopping tablet replica
I20260619 02:47:36.408582 22534 raft_consensus.cc:2229] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:36.409129 22534 raft_consensus.cc:2258] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:36.413388 22534 ts_tablet_manager.cc:1905] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:36.416961 22535 tablet_replica.cc:330] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: stopping tablet replica
I20260619 02:47:36.417399 22535 raft_consensus.cc:2229] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:47:36.418107 22535 raft_consensus.cc:2258] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:36.419430 22533 tablet_replica.cc:330] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: stopping tablet replica
I20260619 02:47:36.419817 22533 raft_consensus.cc:2229] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:36.420298 22533 raft_consensus.cc:2258] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:36.422262 22535 ts_tablet_manager.cc:1905] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:36.422338 22534 ts_tablet_manager.cc:1918] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:36.422500 22534 log.cc:1196] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/015769abd79a44f68dae2e491bcf141f
I20260619 02:47:36.423031 22534 ts_tablet_manager.cc:1939] T 015769abd79a44f68dae2e491bcf141f P c487c33639a44f0886a11ab6f53abd2b: Deleting consensus metadata
I20260619 02:47:36.424853 22021 catalog_manager.cc:4841] TS c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): tablet 015769abd79a44f68dae2e491bcf141f (table test-table [id=486d5334d5f640d1b061f9e82b7131b9]) successfully deleted
Jun 19 02:47:36 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:47:36.429311 22535 ts_tablet_manager.cc:1918] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:36.429514 22535 log.cc:1196] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/015769abd79a44f68dae2e491bcf141f
I20260619 02:47:36.430071 22535 ts_tablet_manager.cc:1939] T 015769abd79a44f68dae2e491bcf141f P d21db929613e48019e782049d140a84a: Deleting consensus metadata
I20260619 02:47:36.431748 22021 catalog_manager.cc:4841] TS d21db929613e48019e782049d140a84a (127.7.246.65:33007): tablet 015769abd79a44f68dae2e491bcf141f (table test-table [id=486d5334d5f640d1b061f9e82b7131b9]) successfully deleted
I20260619 02:47:36.433188 22533 ts_tablet_manager.cc:1905] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:36.434221 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.395638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.7.246.65:33007)
0619 02:47:36.395818 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.395824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.414216 (+ 18392us) server_negotiation.cc:407] Connection header received
0619 02:47:36.414506 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.414512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.414643 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.414731 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:36.418997 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.420154 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.422783 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.423593 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.423690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.428224 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:36.428234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:36.428247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:36.428286 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:36.431410 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.432182 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.432187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.432190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.432321 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.432702 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.432706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.432708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.433010 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:36.433336 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.433648 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.433880 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:47:36.435045 22220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:36.439958 22533 ts_tablet_manager.cc:1918] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:36.440101 22533 log.cc:1196] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/015769abd79a44f68dae2e491bcf141f
I20260619 02:47:36.440539 22533 ts_tablet_manager.cc:1939] T 015769abd79a44f68dae2e491bcf141f P f2353d3c6a7544d891f3495d1451da5d: Deleting consensus metadata
I20260619 02:47:36.442051 22020 catalog_manager.cc:4841] TS f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): tablet 015769abd79a44f68dae2e491bcf141f (table test-table [id=486d5334d5f640d1b061f9e82b7131b9]) successfully deleted
I20260619 02:47:36.538337 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22097
I20260619 02:47:36.556886 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22231
Jun 19 02:47:36 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:47:36.572065 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.557747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.7.246.66:37165)
0619 02:47:36.557917 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.557922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.557939 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:47:36.558039 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.558042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.558108 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.558205 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:36.559492 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.560223 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.561687 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.562053 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.562206 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.566997 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:36.567008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:36.567022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:36.567060 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:36.569728 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.570427 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.570434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.570436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.570557 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.570859 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.570862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.570864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.571207 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:36.571415 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.571666 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.571818 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:47:36.572750 22354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:36.665257 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22231
I20260619 02:47:36.682098 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22364
Jun 19 02:47:36 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:47:36.696868 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:36.682903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.7.246.67:39101)
0619 02:47:36.683105 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:36.683110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:36.683130 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:47:36.683300 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:36.683312 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:36.683384 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:36.683485 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:36.684797 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.685521 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.686877 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:36.687276 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:36.687407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:36.691537 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:36.691551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:36.691586 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:36.691625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:36.694148 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.694796 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.694804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.694806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.694940 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:36.695319 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:36.695325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:36.695329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:36.695660 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:36.695942 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:36.696315 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 02:47:36.696536 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:47:36.697479 22487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:36.784672 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22364
I20260619 02:47:36.800796 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22003
I20260619 02:47:36.801952 22065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:36.912418 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22003
I20260619 02:47:36.933293 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--webserver_interface=127.7.246.126
--webserver_port=43793
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:37.252082 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:37.252398 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:37.252573 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:37.259717 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:37.259856 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:37.259977 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:37.260074 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:37.260159 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:37.269924 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=43793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:37.271195 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:37.272500 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:37.294976 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:37.295017 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:37.294956 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:37.296103 22541 server_base.cc:1034] running on GCE node
I20260619 02:47:37.296988 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:37.298681 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:37.299922 22541 hybrid_clock.cc:648] HybridClock initialized: now 1781837257299876 us; error 47 us; skew 500 ppm
I20260619 02:47:37.300338 22541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:37 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837257, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:37.303879 22541 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:37.305522 22541 webserver.cc:460] Webserver started at http://127.7.246.126:43793/ using document root <none> and password file <none>
I20260619 02:47:37.306308 22541 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:37.306506 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:37.312269 22541 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 02:47:37.315207 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:37.316495 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:47:37.316666 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "481dc8b3c8e34339b7fdb682a647ae6d"
format_stamp: "Formatted at 2026-06-19 02:47:34 on dist-test-slave-9tkz"
server_key: "b0664ff9e4ce47fbbe2e7647b507448c"
server_key_iv: "28d8394e4215c87bed28c83610fdb3fe"
server_key_version: "encryptionkey@0"
I20260619 02:47:37.317795 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:37.361783 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:37.366047 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:37.366320 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:37.385039 22541 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34285
I20260619 02:47:37.385069 22608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34285 every 8 connection(s)
I20260619 02:47:37.386593 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:37.394675 22609 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:47:37.395033 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22541
I20260619 02:47:37.396267 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:33007
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=43433
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:37.400164 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap starting.
I20260619 02:47:37.421658 22609 log.cc:824] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:37.434736 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:47:37.435300 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap complete.
I20260619 02:47:37.445073 22609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:37.446075 22609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Initialized, Role: FOLLOWER
I20260619 02:47:37.446663 22609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:37.446928 22609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:37.447070 22609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:37.447252 22609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:47:37.451231 22609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:37.451799 22609 leader_election.cc:302] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481dc8b3c8e34339b7fdb682a647ae6d; no voters:
I20260619 02:47:37.452467 22609 leader_election.cc:288] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:47:37.452618 22613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:47:37.454061 22613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 LEADER]: Becoming Leader. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Running, Role: LEADER
I20260619 02:47:37.454741 22613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:37.455008 22609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:37.457336 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481dc8b3c8e34339b7fdb682a647ae6d. Latest consensus state: current_term: 2 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:37.457772 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:37.457496 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:37.457964 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:37.470809 22618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:37.479017 22618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:37.480877 22618 catalog_manager.cc:1259] Loaded cluster ID: 364eecc567bc45faa0e3be28c1fbecda
I20260619 02:47:37.481005 22618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:37.485363 22618 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:47:37.485858 22630 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:47:37.497344 22618 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:47:37.504679 22618 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:37.506191 22618 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Loaded TSK: 0
I20260619 02:47:37.507289 22618 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:47:37.757755 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:37.758059 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:37.758255 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:37.765868 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:37.766013 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:37.766178 22611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:37.775784 22611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=43433
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:37.777051 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:37.778373 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:37.791396 22637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:37.791517 22636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:37.792465 22611 server_base.cc:1034] running on GCE node
W20260619 02:47:37.792626 22639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:37.793332 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:37.794314 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:37.795522 22611 hybrid_clock.cc:648] HybridClock initialized: now 1781837257795469 us; error 47 us; skew 500 ppm
I20260619 02:47:37.795822 22611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:37 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837257, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:37.798848 22611 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:37.800486 22611 webserver.cc:460] Webserver started at http://127.7.246.65:43433/ using document root <none> and password file <none>
I20260619 02:47:37.801158 22611 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:37.801314 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:37.806559 22611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:47:37.809314 22646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:37.810837 22611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:47:37.811002 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "d21db929613e48019e782049d140a84a"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "70b375622f5ff42e44fba9108fffed59"
server_key_iv: "2be44f0c110f83b1c5f589072a9f2f49"
server_key_version: "encryptionkey@0"
I20260619 02:47:37.811962 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:37.864435 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:37.868773 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:37.869033 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:37.870102 22611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:37.871496 22611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:37.871634 22611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:37.871781 22611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:37.871866 22611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:37.904760 22611 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33007
I20260619 02:47:37.904872 22758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33007 every 8 connection(s)
I20260619 02:47:37.906312 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:37.911769 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22611
I20260619 02:47:37.912374 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:37165
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=42531
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:37 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837257, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:37.930234 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:37.909399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:37699)
0619 02:47:37.910151 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:37.910193 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:37.911817 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:37.913247 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:37.913256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:37.913894 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:37.914441 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:37.914467 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:37.916261 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:37.916265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:37.917671 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:37.917683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:37.918694 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:37.918701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:37.918989 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:37.919671 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:37.919696 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:37.922225 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:37.925700 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:37.925710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:37.925724 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:37.926388 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:37.926958 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:37.926961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:37.926964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:37.927144 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:37.928109 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:37.928115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:37.928520 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 02:47:37.928747 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 02:47:37.929130 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":298,"threads_started":1}
I20260619 02:47:37.930900 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:37.909210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:37699 (local address 127.7.246.126:34285)
0619 02:47:37.909885 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:37.909901 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:37.910315 (+ 414us) server_negotiation.cc:407] Connection header received
0619 02:47:37.912347 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:37.912358 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:37.912752 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:37.913100 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:37.914691 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:37.916097 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:37.917808 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:37.918458 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:37.918659 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:37.922520 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:37.922529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:37.922542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:37.922581 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:37.925497 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:37.926578 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:37.926590 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:37.926600 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:37.926814 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:37.927271 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:37.927279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:37.927285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:37.927947 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:37.928158 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:37.929041 (+ 883us) server_negotiation.cc:299] Negotiation successful
0619 02:47:37.929509 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":240,"threads_started":1}
I20260619 02:47:37.933596 22759 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:37.933913 22759 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:37.934613 22759 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:37.936684 22574 ts_manager.cc:194] Registered new tserver with Master: d21db929613e48019e782049d140a84a (127.7.246.65:33007)
I20260619 02:47:37.938222 22574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:37699
W20260619 02:47:38.230729 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:38.231040 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:38.231215 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:38.238397 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:38.238534 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:38.238716 22763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:38.248051 22763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:37165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=42531
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:38.249343 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:38.250644 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:38.263170 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:38.263236 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:38.263211 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:38.264384 22763 server_base.cc:1034] running on GCE node
I20260619 02:47:38.265045 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:38.266028 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:38.267253 22763 hybrid_clock.cc:648] HybridClock initialized: now 1781837258267199 us; error 41 us; skew 500 ppm
I20260619 02:47:38.267562 22763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:38 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837258, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:38.270572 22763 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:38.271883 22763 webserver.cc:460] Webserver started at http://127.7.246.66:42531/ using document root <none> and password file <none>
I20260619 02:47:38.272568 22763 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:38.272725 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:38.277820 22763 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:47:38.280531 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.281924 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:38.282119 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "c487c33639a44f0886a11ab6f53abd2b"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "d1b9033ca4a6d658be546bdec2dbd600"
server_key_iv: "06fdec810891f418e7f745495ad58a34"
server_key_version: "encryptionkey@0"
I20260619 02:47:38.283104 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:38.298267 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:38.302038 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:38.302304 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:38.303221 22763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:38.304512 22763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:38.304638 22763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.304771 22763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:38.304862 22763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.337129 22763 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:37165
I20260619 02:47:38.337214 22890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:37165 every 8 connection(s)
I20260619 02:47:38.338676 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:38.345588 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22763
I20260619 02:47:38.346189 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:39101
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=46495
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:47:38 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837258, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:38.358124 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.341138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44735 (local address 127.7.246.126:34285)
0619 02:47:38.341314 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.341318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.342526 (+ 1208us) server_negotiation.cc:407] Connection header received
0619 02:47:38.344290 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.344295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.344399 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.344516 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:38.346031 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.346793 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.348275 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.348698 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.348828 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.352369 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:38.352373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:38.352376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:38.352406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:38.355139 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.355858 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.355870 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.355873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.356007 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.356449 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.356465 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.356467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.356715 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:38.356869 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.357535 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.357780 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:47:38.359053 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.341648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:44735)
0619 02:47:38.342336 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:38.342394 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:38.344031 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:38.344743 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:38.344751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:38.345232 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:38.345867 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.345885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.346951 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.346956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.348127 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.348138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.348911 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.348918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.349146 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.349881 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:38.349902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:38.352152 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:38.355292 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:38.355300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:38.355321 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:38.355703 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:38.356140 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:38.356145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:38.356151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:38.356328 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:38.356844 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:38.356852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:38.357167 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 02:47:38.357417 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 02:47:38.357915 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260619 02:47:38.360457 22891 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:38.360766 22891 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:38.361644 22891 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:38.363263 22574 ts_manager.cc:194] Registered new tserver with Master: c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:38.364174 22574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:44735
W20260619 02:47:38.676504 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:38.676779 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:38.676939 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:38.684182 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:38.684275 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:38.684383 22894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:47:38.694006 22894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:39101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=46495
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:38.695154 22894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:38.696317 22894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:38.709277 22899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:38.709240 22900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:38.709520 22902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:38.710157 22894 server_base.cc:1034] running on GCE node
I20260619 02:47:38.710775 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:38.711715 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:38.712939 22894 hybrid_clock.cc:648] HybridClock initialized: now 1781837258712882 us; error 53 us; skew 500 ppm
I20260619 02:47:38.713251 22894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:38 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837258, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:38.716317 22894 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:47:38.717773 22894 webserver.cc:460] Webserver started at http://127.7.246.67:46495/ using document root <none> and password file <none>
I20260619 02:47:38.718408 22894 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:38.718513 22894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:38.723594 22894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 02:47:38.726052 22909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.727308 22894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:47:38.727468 22894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "f2353d3c6a7544d891f3495d1451da5d"
format_stamp: "Formatted at 2026-06-19 02:47:36 on dist-test-slave-9tkz"
server_key: "1ba071acf59e81524c044a161330baf9"
server_key_iv: "1a80da62fe61d1228f22bbd465bffd54"
server_key_version: "encryptionkey@0"
I20260619 02:47:38.728387 22894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:38.746805 22894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:38.750563 22894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:38.750888 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:38.751852 22894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:38.753134 22894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:38.753226 22894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.753377 22894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:38.753466 22894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:38.787156 22894 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39101
I20260619 02:47:38.787177 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39101 every 8 connection(s)
I20260619 02:47:38.788656 22894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:47:38.789162 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 22894
Jun 19 02:47:38 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837258, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:38.805342 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.790396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.7.246.126:34285)
0619 02:47:38.790596 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.790603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.791007 (+ 404us) server_negotiation.cc:407] Connection header received
0619 02:47:38.791207 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.791212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.791291 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.791387 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:38.793442 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.794601 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.795889 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.796547 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.796671 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.799234 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:38.799240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:38.799245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:38.799282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:38.802771 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.803376 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.803382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.803385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.803518 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.803901 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.803904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.803906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.804238 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:38.804418 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.804771 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.804960 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:47:38.811110 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.791346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:50967 (local address 127.7.246.126:34285)
0619 02:47:38.791831 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.791835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.792999 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 02:47:38.795363 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.795368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.795450 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.795575 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:38.797596 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.798331 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.800342 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.800856 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.801009 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.805030 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:38.805036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:38.805040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:38.805072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:38.808478 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.809113 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.809117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.809119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.809221 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:38.809618 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:38.809623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:38.809625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:38.809976 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:38.810092 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.810694 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.810833 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":296,"threads_started":1}
I20260619 02:47:38.812116 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.792056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:50967)
0619 02:47:38.792831 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:38.792866 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:38.795096 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:38.795847 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:38.795861 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:38.796584 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:38.797405 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.797426 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.798489 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.798494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.800183 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.800198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.801124 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.801135 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.801464 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.802081 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:38.802118 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:38.804807 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:38.808593 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:38.808599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:38.808611 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:38.809010 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:38.809309 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:38.809312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:38.809314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:38.809477 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:38.810094 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:38.810099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:38.810393 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:47:38.810629 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 02:47:38.810986 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":325,"threads_started":1}
I20260619 02:47:38.813345 23022 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34285
I20260619 02:47:38.813683 23022 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:38.814370 23022 heartbeater.cc:511] Master 127.7.246.126:34285 requested a full tablet report, sending...
I20260619 02:47:38.815723 22574 ts_manager.cc:194] Registered new tserver with Master: f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101)
I20260619 02:47:38.816491 22574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:50967
I20260619 02:47:38.820808 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:47:38.827457 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.821713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.7.246.126:34285)
0619 02:47:38.821879 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.821883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.822392 (+ 509us) server_negotiation.cc:407] Connection header received
0619 02:47:38.822622 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.822627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.822759 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.822901 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:47:38.823323 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.823999 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.825419 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.825878 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.826016 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.826490 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:47:38.826902 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:47:38.827059 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.827152 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.827224 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:47:38.828199 22574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:47:38.835471 22574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:47:38.863651 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.846790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.7.246.67:39101)
0619 02:47:38.851300 (+ 4510us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.851311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.851356 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:47:38.851487 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.851497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.851807 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.852925 (+ 1118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:38.853256 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.856717 (+ 3461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.860827 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.862258 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.862425 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.862685 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.863145 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.863302 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3113,"thread_start_us":2946,"threads_started":1}
I20260619 02:47:38.863803 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.846432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39101 (local address 127.0.0.1:47164)
0619 02:47:38.848690 (+ 2258us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:38.848735 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:38.848919 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:38.852097 (+ 3178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:38.852109 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:38.852158 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:38.852663 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.852679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.856906 (+ 4227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.856915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.859597 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.859612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.862551 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.862557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.862808 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.862943 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 02:47:38.863090 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:47:38.863324 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"mutex_wait_us":44,"thread_start_us":15207,"threads_started":1}
I20260619 02:47:38.867408 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.846910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42186)
0619 02:47:38.858046 (+ 11136us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:38.858065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:38.858764 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:38.859953 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:38.859958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:38.859985 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:38.860399 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.860412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.862360 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.862365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.864491 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.864504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.866542 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.866548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.866736 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.866767 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:47:38.866882 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 02:47:38.867005 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11004,"mutex_wait_us":1,"thread_start_us":1332,"threads_started":1}
I20260619 02:47:38.867427 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.847529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.7.246.65:33007)
0619 02:47:38.849561 (+ 2032us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.849573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.858186 (+ 8613us) server_negotiation.cc:407] Connection header received
0619 02:47:38.858952 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.858961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.859192 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.859349 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:38.860602 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.862220 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.864652 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.866285 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.866448 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.866658 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.866936 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.867098 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1847,"thread_start_us":1716,"threads_started":1}
I20260619 02:47:38.867928 22957 tablet_service.cc:1467] Processing CreateTablet for tablet df15952db4884abe9b4799843f55f8e4 (DEFAULT_TABLE table=test-table [id=74186da9221340329e7f500052b011f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:38.869284 22957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df15952db4884abe9b4799843f55f8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:38.869910 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.849246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37165 (local address 127.0.0.1:57048)
0619 02:47:38.861887 (+ 12641us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:38.861902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:38.862036 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:38.862775 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:38.862779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:38.862804 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:38.863308 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.863320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.865428 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.865433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.867466 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:38.867482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.869103 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.869109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:38.869274 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.869307 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:47:38.869392 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:47:38.869514 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12534,"mutex_wait_us":3,"thread_start_us":9304,"threads_started":1}
I20260619 02:47:38.870404 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:38.849947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.7.246.66:37165)
0619 02:47:38.861483 (+ 11536us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:38.861494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:38.862000 (+ 506us) server_negotiation.cc:407] Connection header received
0619 02:47:38.862200 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:38.862206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:38.862421 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:38.862625 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:38.863527 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.865263 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.867656 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:38.868885 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:38.869052 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:38.869232 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:38.869684 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 02:47:38.869840 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11392,"thread_start_us":11283,"threads_started":1}
I20260619 02:47:38.871037 22694 tablet_service.cc:1467] Processing CreateTablet for tablet df15952db4884abe9b4799843f55f8e4 (DEFAULT_TABLE table=test-table [id=74186da9221340329e7f500052b011f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:38.872377 22694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df15952db4884abe9b4799843f55f8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:38.873258 22826 tablet_service.cc:1467] Processing CreateTablet for tablet df15952db4884abe9b4799843f55f8e4 (DEFAULT_TABLE table=test-table [id=74186da9221340329e7f500052b011f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:47:38.874303 22826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df15952db4884abe9b4799843f55f8e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:38.884614 23038 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Bootstrap starting.
I20260619 02:47:38.887316 23040 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Bootstrap starting.
I20260619 02:47:38.888090 23039 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Bootstrap starting.
I20260619 02:47:38.889988 23038 tablet_bootstrap.cc:654] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:38.890802 23040 tablet_bootstrap.cc:654] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:38.891085 23038 log.cc:824] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:38.891444 23039 tablet_bootstrap.cc:654] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:38.892031 23040 log.cc:824] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:38.893013 23039 log.cc:824] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:38.894340 23038 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: No bootstrap required, opened a new log
I20260619 02:47:38.894625 23038 ts_tablet_manager.cc:1397] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 02:47:38.895251 23040 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: No bootstrap required, opened a new log
I20260619 02:47:38.895571 23040 ts_tablet_manager.cc:1397] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 02:47:38.896257 23039 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: No bootstrap required, opened a new log
I20260619 02:47:38.896493 23039 ts_tablet_manager.cc:1397] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 02:47:38.901401 23038 raft_consensus.cc:348] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.901809 23038 raft_consensus.cc:374] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:38.901906 23038 raft_consensus.cc:729] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2353d3c6a7544d891f3495d1451da5d, State: Initialized, Role: FOLLOWER
I20260619 02:47:38.902503 23038 consensus_queue.cc:260] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.904119 23022 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:38.906210 23038 ts_tablet_manager.cc:1428] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 02:47:38.906626 23039 raft_consensus.cc:348] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.907038 23039 raft_consensus.cc:374] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:38.907155 23039 raft_consensus.cc:729] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c487c33639a44f0886a11ab6f53abd2b, State: Initialized, Role: FOLLOWER
I20260619 02:47:38.907850 23039 consensus_queue.cc:260] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.908762 23040 raft_consensus.cc:348] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.909196 23040 raft_consensus.cc:374] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:38.909330 23040 raft_consensus.cc:729] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21db929613e48019e782049d140a84a, State: Initialized, Role: FOLLOWER
I20260619 02:47:38.910043 23040 consensus_queue.cc:260] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:38.911181 22891 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:38.912878 23039 ts_tablet_manager.cc:1428] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260619 02:47:38.914252 22759 heartbeater.cc:503] Master 127.7.246.126:34285 was elected leader, sending a full tablet report...
I20260619 02:47:38.914709 23040 ts_tablet_manager.cc:1428] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260619 02:47:38.999646 23046 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:39.000021 23046 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:39.001974 23046 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:39.009375 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.002494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:59937 (local address 127.7.246.66:37165)
0619 02:47:39.002726 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.002747 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.003754 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 02:47:39.003954 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.003960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.004045 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.004137 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:39.004736 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.005406 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.007236 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.008461 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.008610 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.008688 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.008982 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.009115 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:47:39.009373 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.002405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:38689 (local address 127.7.246.67:39101)
0619 02:47:39.002778 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.002787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.003648 (+ 861us) server_negotiation.cc:407] Connection header received
0619 02:47:39.003859 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.003863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.003920 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.004036 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:47:39.004563 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.005299 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.007108 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.008344 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.008530 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.008609 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.008982 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.009115 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260619 02:47:39.009497 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.002428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37165 (local address 127.7.246.65:59937)
0619 02:47:39.003672 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.003689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:39.003820 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:39.004179 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:39.004186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:39.004223 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:39.004583 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:39.004591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.005538 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.005582 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:39.006996 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:39.007009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.008663 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.008671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:39.008805 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.008837 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:47:39.008916 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:47:39.009037 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":1,"thread_start_us":572,"threads_started":1}
I20260619 02:47:39.009518 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.002383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39101 (local address 127.7.246.65:38689)
0619 02:47:39.003486 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.003507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:39.003689 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:39.004051 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:39.004055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:39.004078 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:47:39.004417 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:39.004425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.005435 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.005440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:39.006926 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:39.006938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.008555 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.008562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:39.008755 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.008792 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:47:39.008893 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:47:39.009043 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":30,"thread_start_us":953,"threads_started":1}
I20260619 02:47:39.010517 22977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df15952db4884abe9b4799843f55f8e4" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2353d3c6a7544d891f3495d1451da5d" is_pre_election: true
I20260619 02:47:39.010792 22846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df15952db4884abe9b4799843f55f8e4" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c487c33639a44f0886a11ab6f53abd2b" is_pre_election: true
I20260619 02:47:39.010985 22977 raft_consensus.cc:2454] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 0.
I20260619 02:47:39.011266 22846 raft_consensus.cc:2454] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 0.
I20260619 02:47:39.011606 22648 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d21db929613e48019e782049d140a84a, f2353d3c6a7544d891f3495d1451da5d; no voters:
I20260619 02:47:39.011976 23046 raft_consensus.cc:2781] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:47:39.012132 23046 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:47:39.012233 23046 raft_consensus.cc:3037] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:39.014394 23046 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:39.014948 23046 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 election: Requested vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
I20260619 02:47:39.015338 22977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df15952db4884abe9b4799843f55f8e4" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2353d3c6a7544d891f3495d1451da5d"
I20260619 02:47:39.015432 22846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df15952db4884abe9b4799843f55f8e4" candidate_uuid: "d21db929613e48019e782049d140a84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c487c33639a44f0886a11ab6f53abd2b"
I20260619 02:47:39.015542 22977 raft_consensus.cc:3037] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:39.015651 22846 raft_consensus.cc:3037] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:39.018800 22846 raft_consensus.cc:2454] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 1.
I20260619 02:47:39.018800 22977 raft_consensus.cc:2454] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21db929613e48019e782049d140a84a in term 1.
I20260619 02:47:39.019181 22648 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d21db929613e48019e782049d140a84a, f2353d3c6a7544d891f3495d1451da5d; no voters:
I20260619 02:47:39.019464 23046 raft_consensus.cc:2781] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:39.019920 23046 raft_consensus.cc:686] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 LEADER]: Becoming Leader. State: Replica: d21db929613e48019e782049d140a84a, State: Running, Role: LEADER
I20260619 02:47:39.020530 23046 consensus_queue.cc:237] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:39.025116 22574 catalog_manager.cc:5495] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a reported cstate change: term changed from 0 to 1, leader changed from <none> to d21db929613e48019e782049d140a84a (127.7.246.65). New cstate: current_term: 1 leader_uuid: "d21db929613e48019e782049d140a84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } health_report { overall_health: UNKNOWN } } }
W20260619 02:47:39.040386 23023 tablet.cc:2369] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:47:39.056293 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.050824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.7.246.65:33007)
0619 02:47:39.051020 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.051028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.051051 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:47:39.051304 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.051308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.051379 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.051531 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:47:39.051908 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.052593 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.054078 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.054507 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.054639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.055148 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:47:39.055723 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:47:39.055898 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.055949 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.056026 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:47:39.064638 22977 raft_consensus.cc:1261] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Refusing update from remote peer d21db929613e48019e782049d140a84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:39.064656 22846 raft_consensus.cc:1261] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Refusing update from remote peer d21db929613e48019e782049d140a84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:47:39.065737 23049 consensus_queue.cc:1035] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:39.066112 23046 consensus_queue.cc:1035] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:47:39.092836 22573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:39.093166 22573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:47:39.096735 22573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Sending DeleteTablet for 3 replicas of tablet df15952db4884abe9b4799843f55f8e4
I20260619 02:47:39.097870 22957 tablet_service.cc:1514] Processing DeleteTablet for tablet df15952db4884abe9b4799843f55f8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:39 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47164
I20260619 02:47:39.098102 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22611
I20260619 02:47:39.098095 22826 tablet_service.cc:1514] Processing DeleteTablet for tablet df15952db4884abe9b4799843f55f8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:39 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:57048
I20260619 02:47:39.098320 22694 tablet_service.cc:1514] Processing DeleteTablet for tablet df15952db4884abe9b4799843f55f8e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:47:39 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:42186
I20260619 02:47:39.106664 23062 tablet_replica.cc:330] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: stopping tablet replica
I20260619 02:47:39.107136 23062 raft_consensus.cc:2229] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:39.107697 23062 raft_consensus.cc:2258] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:39.109747 23062 ts_tablet_manager.cc:1905] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:39.110142 23061 tablet_replica.cc:330] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: stopping tablet replica
I20260619 02:47:39.110661 23061 raft_consensus.cc:2229] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:47:39.111616 23061 raft_consensus.cc:2258] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:47:39.116086 23061 ts_tablet_manager.cc:1905] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:47:39.117377 23062 ts_tablet_manager.cc:1918] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:39.117628 23062 log.cc:1196] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/df15952db4884abe9b4799843f55f8e4
I20260619 02:47:39.117681 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.099241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.7.246.65:33007)
0619 02:47:39.099461 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.099467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.099482 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:47:39.099528 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.099533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.099622 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.099731 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:39.101900 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.102766 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.105169 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.105696 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.105828 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.110045 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:39.110057 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:39.110074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:39.110126 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:39.114129 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.115179 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.115188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.115193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.115374 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.115835 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.115840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.115843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.116304 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:39.116615 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.117014 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.117250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:47:39.118353 22754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:39.118558 23062 ts_tablet_manager.cc:1939] T df15952db4884abe9b4799843f55f8e4 P c487c33639a44f0886a11ab6f53abd2b: Deleting consensus metadata
I20260619 02:47:39.120498 22560 catalog_manager.cc:4841] TS c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): tablet df15952db4884abe9b4799843f55f8e4 (table test-table [id=74186da9221340329e7f500052b011f0]) successfully deleted
I20260619 02:47:39.126220 23061 ts_tablet_manager.cc:1918] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:47:39.126390 23061 log.cc:1196] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/df15952db4884abe9b4799843f55f8e4
I20260619 02:47:39.126843 23061 ts_tablet_manager.cc:1939] T df15952db4884abe9b4799843f55f8e4 P f2353d3c6a7544d891f3495d1451da5d: Deleting consensus metadata
I20260619 02:47:39.128265 22559 catalog_manager.cc:4841] TS f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): tablet df15952db4884abe9b4799843f55f8e4 (table test-table [id=74186da9221340329e7f500052b011f0]) successfully deleted
I20260619 02:47:39.216455 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22611
W20260619 02:47:39.234385 22560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:33007 (error 108)
I20260619 02:47:39.234563 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22763
I20260619 02:47:39.236330 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.235829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42216)
0619 02:47:39.236008 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.236143 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:47:39.236608 22560 catalog_manager.cc:4584] TS d21db929613e48019e782049d140a84a (127.7.246.65:33007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df15952db4884abe9b4799843f55f8e4: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
I20260619 02:47:39.248116 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.235363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.7.246.66:37165)
0619 02:47:39.235635 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.235644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.235670 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:47:39.235814 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.235820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.235921 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.236057 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:39.237513 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.238341 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.239544 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.239949 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.240048 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.242501 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:39.242515 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:39.242529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:39.242563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:39.245777 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.246463 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.246471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.246473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.246600 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.246905 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.246908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.246910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.247257 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:39.247502 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.247689 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.247850 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 02:47:39.248764 22886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:39.290242 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.289775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42222)
0619 02:47:39.289979 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.290078 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 02:47:39.291287 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.290942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42224)
0619 02:47:39.291100 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.291168 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:47:39.338428 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22763
I20260619 02:47:39.351804 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.351356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42232)
0619 02:47:39.351553 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.351650 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:47:39.352864 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.352507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42236)
0619 02:47:39.352695 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.352758 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 02:47:39.356235 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22894
I20260619 02:47:39.373638 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.361189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.7.246.67:39101)
0619 02:47:39.361364 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:39.361369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:39.361390 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:47:39.361591 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:39.361599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:39.361696 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:39.361798 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:39.363145 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.363854 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.365293 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:39.365737 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:39.365823 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:39.368215 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:39.368228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:39.368244 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:39.368277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:39.371124 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.371711 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.371720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.371722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.371906 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:39.372266 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:39.372270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:39.372272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:39.372649 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:39.372880 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:39.373145 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:47:39.373302 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:47:39.374264 23017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:39.433332 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.432912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42252)
0619 02:47:39.433104 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.433191 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 02:47:39.434428 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:39.434081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:33007 (local address 127.0.0.1:42262)
0619 02:47:39.434259 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:39.434325 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 02:47:39.464927 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22894
I20260619 02:47:39.481397 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 22541
I20260619 02:47:39.482769 22604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:39.593286 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 22541
I20260619 02:47:39.614452 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--webserver_interface=127.7.246.126
--webserver_port=43793
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:39.927882 23069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:39.928143 23069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:39.928299 23069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:39.935382 23069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:39.935524 23069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:39.935603 23069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:39.935705 23069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:39.935767 23069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:39.945106 23069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=43793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:39.946367 23069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:39.948030 23069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:39.959124 23077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:39.959087 23075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:39.959160 23074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:39.959897 23069 server_base.cc:1034] running on GCE node
I20260619 02:47:39.960688 23069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:39.962080 23069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:39.963306 23069 hybrid_clock.cc:648] HybridClock initialized: now 1781837259963261 us; error 29 us; skew 500 ppm
I20260619 02:47:39.963634 23069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:39 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837259, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:39.970666 23069 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:39.972118 23069 webserver.cc:460] Webserver started at http://127.7.246.126:43793/ using document root <none> and password file <none>
I20260619 02:47:39.972743 23069 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:39.972841 23069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:39.977980 23069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:47:39.980681 23084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:39.982066 23069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:47:39.982239 23069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "481dc8b3c8e34339b7fdb682a647ae6d"
format_stamp: "Formatted at 2026-06-19 02:47:34 on dist-test-slave-9tkz"
server_key: "b0664ff9e4ce47fbbe2e7647b507448c"
server_key_iv: "28d8394e4215c87bed28c83610fdb3fe"
server_key_version: "encryptionkey@0"
I20260619 02:47:39.983207 23069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:40.012612 23069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:40.016515 23069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:40.016793 23069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:40.034937 23069 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34285
I20260619 02:47:40.034960 23135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34285 every 8 connection(s)
I20260619 02:47:40.036345 23069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:40.042855 23136 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:47:40.044687 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23069
I20260619 02:47:40.045487 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:33007
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=43433
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:40.046224 23136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap starting.
I20260619 02:47:40.073343 23136 log.cc:824] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:40.099363 23136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:47:40.100126 23136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d: Bootstrap complete.
I20260619 02:47:40.113523 23136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:40.114550 23136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Initialized, Role: FOLLOWER
I20260619 02:47:40.115164 23136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:40.115427 23136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:40.115577 23136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:40.115792 23136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:47:40.119761 23136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:40.120285 23136 leader_election.cc:302] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 481dc8b3c8e34339b7fdb682a647ae6d; no voters:
I20260619 02:47:40.120985 23136 leader_election.cc:288] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:47:40.121148 23140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:47:40.122499 23140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [term 3 LEADER]: Becoming Leader. State: Replica: 481dc8b3c8e34339b7fdb682a647ae6d, State: Running, Role: LEADER
I20260619 02:47:40.123061 23140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } }
I20260619 02:47:40.123337 23136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:40.126124 23141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:40.126603 23141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:40.131124 23142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 481dc8b3c8e34339b7fdb682a647ae6d. Latest consensus state: current_term: 3 leader_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "481dc8b3c8e34339b7fdb682a647ae6d" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34285 } } }
I20260619 02:47:40.131474 23142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 481dc8b3c8e34339b7fdb682a647ae6d [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:40.133634 23146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:40.142205 23146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:40.143697 23146 catalog_manager.cc:1259] Loaded cluster ID: 364eecc567bc45faa0e3be28c1fbecda
I20260619 02:47:40.143837 23146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:47:40.145869 23146 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc407f2f9ad google::LogMessage::Fail() at ??:0
@ 0x7fc407f3288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc407f2f4de google::LogMessage::Flush() at ??:0
@ 0x7fc407f30a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc41d962512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc41d95f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc41d9e7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc41d9e7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc40ab7b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc40ab7e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20260619 02:47:40.197887 23159 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc40ab7e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc40ab59eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc40de336db start_thread at ??:0
@ 0x7fc406dc961f clone at ??:0
W20260619 02:47:40.448247 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:40.448496 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:40.448589 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:40.455708 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:40.455801 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:40.455905 23138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:40.465153 23138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=43433
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:40.466364 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:40.467541 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:40.481616 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:40.481670 23177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:40.481572 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:40.483635 23138 server_base.cc:1034] running on GCE node
I20260619 02:47:40.484274 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:40.485205 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:40.486438 23138 hybrid_clock.cc:648] HybridClock initialized: now 1781837260486393 us; error 62 us; skew 500 ppm
I20260619 02:47:40.486761 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:40 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837260, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:40.489903 23138 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:40.491250 23138 webserver.cc:460] Webserver started at http://127.7.246.65:43433/ using document root <none> and password file <none>
I20260619 02:47:40.491894 23138 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:40.491987 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:40.497077 23138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:40.499811 23187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:40.501051 23138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:40.501217 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "d21db929613e48019e782049d140a84a"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "70b375622f5ff42e44fba9108fffed59"
server_key_iv: "2be44f0c110f83b1c5f589072a9f2f49"
server_key_version: "encryptionkey@0"
I20260619 02:47:40.502192 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:40.515769 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:40.519843 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:40.520119 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:40.521046 23138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:40.522822 23194 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 02:47:40.528192 23138 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:47:40.528344 23138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260619 02:47:40.528481 23138 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:47:40.531421 23138 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:47:40.531528 23138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.001s
I20260619 02:47:40.531898 23194 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Bootstrap starting.
I20260619 02:47:40.559629 23194 log.cc:824] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:40.571626 23138 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:33007
I20260619 02:47:40.571674 23301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:33007 every 8 connection(s)
I20260619 02:47:40.572642 23194 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:47:40.572800 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:40.573190 23194 tablet_bootstrap.cc:492] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Bootstrap complete.
I20260619 02:47:40.574882 23194 ts_tablet_manager.cc:1397] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.017s
I20260619 02:47:40.577877 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:40.575511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:45349)
0619 02:47:40.576363 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:40.576669 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":336,"threads_started":1}
W20260619 02:47:40.578519 23302 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34285 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34285: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
I20260619 02:47:40.579409 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:40.579012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:49039)
0619 02:47:40.579204 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:40.579284 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:47:40.579797 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23138
I20260619 02:47:40.580433 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:37165
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=42531
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:40.583359 23194 raft_consensus.cc:348] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:40.584800 23194 raft_consensus.cc:729] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21db929613e48019e782049d140a84a, State: Initialized, Role: FOLLOWER
I20260619 02:47:40.585533 23194 consensus_queue.cc:260] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:40.587549 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:40.585948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:38921)
0619 02:47:40.587175 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:40.587312 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1082}
W20260619 02:47:40.588047 23302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:47:40.589387 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:40.588986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:33133)
0619 02:47:40.589176 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:40.589250 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 02:47:40.589489 23194 ts_tablet_manager.cc:1428] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 02:47:40.594504 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:40.594210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:38127)
0619 02:47:40.594337 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:40.594399 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:40.912168 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:40.912443 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:40.912544 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:40.919777 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:40.919880 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:40.919997 23306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:40.929333 23306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:37165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=42531
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:40.930647 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:40.931929 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:40.945039 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:40.945058 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:40.945428 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:40.946044 23306 server_base.cc:1034] running on GCE node
I20260619 02:47:40.946647 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:40.947568 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:40.948805 23306 hybrid_clock.cc:648] HybridClock initialized: now 1781837260948765 us; error 50 us; skew 500 ppm
I20260619 02:47:40.949127 23306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:40 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837260, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:40.952154 23306 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:40.953747 23306 webserver.cc:460] Webserver started at http://127.7.246.66:42531/ using document root <none> and password file <none>
I20260619 02:47:40.954449 23306 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:40.954603 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:40.959770 23306 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:47:40.963949 23321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:40.965253 23306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:40.965435 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "c487c33639a44f0886a11ab6f53abd2b"
format_stamp: "Formatted at 2026-06-19 02:47:35 on dist-test-slave-9tkz"
server_key: "d1b9033ca4a6d658be546bdec2dbd600"
server_key_iv: "06fdec810891f418e7f745495ad58a34"
server_key_version: "encryptionkey@0"
I20260619 02:47:40.966492 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:40.979315 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:40.983019 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:40.983278 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:40.984206 23306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:40.985445 23306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:40.985630 23306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:40.985755 23306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:40.985847 23306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:41.018826 23306 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:37165
I20260619 02:47:41.018909 23433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:37165 every 8 connection(s)
I20260619 02:47:41.020310 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:41.024861 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.023183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:33071)
0619 02:47:41.023874 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.024092 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":287,"threads_started":1}
I20260619 02:47:41.025446 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23306
W20260619 02:47:41.025506 23434 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34285 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34285: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
I20260619 02:47:41.026069 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:39101
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:34285
--webserver_port=46495
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:47:41.026347 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.025985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:56487)
0619 02:47:41.026124 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.026214 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:47:41.027176 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.026862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:59761)
0619 02:47:41.027000 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.027079 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:47:41.027366 23434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:47:41.342372 23437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:41.342685 23437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:41.342773 23437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:41.349937 23437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:41.350039 23437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:41.350150 23437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:47:41.359439 23437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:39101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=46495
--tserver_master_addrs=127.7.246.126:34285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:41.360579 23437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:41.361783 23437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:41.374464 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:41.374380 23445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:41.374434 23443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:41.375216 23437 server_base.cc:1034] running on GCE node
I20260619 02:47:41.375703 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:41.376507 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:41.377710 23437 hybrid_clock.cc:648] HybridClock initialized: now 1781837261377651 us; error 49 us; skew 500 ppm
I20260619 02:47:41.378011 23437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:41 dist-test-slave-9tkz krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837261, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:41.381001 23437 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:47:41.382354 23437 webserver.cc:460] Webserver started at http://127.7.246.67:46495/ using document root <none> and password file <none>
I20260619 02:47:41.383011 23437 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:41.383141 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:41.388180 23437 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:47:41.390682 23452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:41.391949 23437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:47:41.392138 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "f2353d3c6a7544d891f3495d1451da5d"
format_stamp: "Formatted at 2026-06-19 02:47:36 on dist-test-slave-9tkz"
server_key: "1ba071acf59e81524c044a161330baf9"
server_key_iv: "1a80da62fe61d1228f22bbd465bffd54"
server_key_version: "encryptionkey@0"
I20260619 02:47:41.393070 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:41.411713 23437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:41.415400 23437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:41.415678 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:41.416589 23437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:41.417968 23437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:41.418087 23437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:41.418221 23437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:41.418311 23437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:41.450688 23437 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39101
I20260619 02:47:41.450769 23564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39101 every 8 connection(s)
I20260619 02:47:41.452090 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:47:41.456499 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.454869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:51483)
0619 02:47:41.455526 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.455736 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":267,"threads_started":1}
W20260619 02:47:41.457083 23565 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34285 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34285: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
I20260619 02:47:41.457836 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.457512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:33499)
0619 02:47:41.457669 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.457745 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:47:41.458556 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.458293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:52171)
0619 02:47:41.458398 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.458461 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 02:47:41.458832 23565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:47:41.459080 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23437
I20260619 02:47:41.596061 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:41.595337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:57067)
0619 02:47:41.595766 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:41.595842 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":201,"threads_started":1}
I20260619 02:47:42.028612 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.027850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:38199)
0619 02:47:42.028294 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:42.028386 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260619 02:47:42.287271 23571 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:42.287569 23571 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:42.289177 23571 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
Jun 19 02:47:42 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837260, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
Jun 19 02:47:42 dist-test-slave-9tkz krb5kdc[21990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837260, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:47:42.323560 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.289960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:49991 (local address 127.7.246.66:37165)
0619 02:47:42.302217 (+ 12257us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:42.302229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:42.302559 (+ 330us) server_negotiation.cc:407] Connection header received
0619 02:47:42.304395 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:42.304402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:42.304708 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:42.305015 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:42.307060 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.308486 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.310302 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.310899 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.311094 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:42.315053 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:42.315060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:42.315072 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:42.315108 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:42.318994 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320136 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:42.320143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:42.320149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:42.320324 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320837 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:42.320843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:42.320846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:42.321475 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:42.321710 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:42.322755 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0619 02:47:42.323155 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12083,"thread_start_us":11924,"threads_started":1}
I20260619 02:47:42.323583 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.289702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:49977 (local address 127.7.246.67:39101)
0619 02:47:42.301568 (+ 11866us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:42.301589 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:42.302103 (+ 514us) server_negotiation.cc:407] Connection header received
0619 02:47:42.304247 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:42.304255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:42.304725 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:42.305120 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:42.307054 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.308459 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.310300 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.310908 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.311105 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:42.315412 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:42.315421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:42.315436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:42.315483 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:42.319044 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320129 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:42.320137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:42.320145 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:42.320324 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320837 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:42.320843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:42.320846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:42.321471 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:42.321711 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:42.322755 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0619 02:47:42.323192 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11655,"thread_start_us":11535,"threads_started":1}
I20260619 02:47:42.323619 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.289404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39101 (local address 127.7.246.65:49977)
0619 02:47:42.301898 (+ 12494us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:42.301941 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:42.302564 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:42.305121 (+ 2557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:42.305131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:42.306140 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:42.306861 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:42.306879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.308646 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.308650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:42.310169 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:42.310181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.311137 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.311145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:42.311449 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:42.312178 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:42.312204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:42.315136 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:42.319168 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:42.319178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:42.319193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:42.319993 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:42.320441 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:42.320450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:42.320715 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:42.321665 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:42.321674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:42.322151 (+ 477us) client_negotiation.cc:769] Sending connection context
0619 02:47:42.322480 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 02:47:42.323037 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12361,"mutex_wait_us":288,"spinlock_wait_cycles":32640,"thread_start_us":12216,"threads_started":1}
I20260619 02:47:42.323645 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.289690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37165 (local address 127.7.246.65:49991)
0619 02:47:42.302427 (+ 12737us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:42.302441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:42.302596 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:42.305123 (+ 2527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:42.305132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:42.306136 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:42.306860 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:42.306878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.308657 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.308661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:42.310174 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:42.310185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:42.311107 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:42.311117 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:42.311453 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:42.312245 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:42.312261 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:42.314776 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:42.319168 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:42.319178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:42.319193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:42.319983 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:42.320441 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:42.320446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:42.320450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:42.320714 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:42.321665 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:42.321674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:42.322151 (+ 477us) client_negotiation.cc:769] Sending connection context
0619 02:47:42.322503 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 02:47:42.323169 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12646,"mutex_wait_us":450,"spinlock_wait_cycles":15744,"thread_start_us":11914,"threads_started":1}
W20260619 02:47:42.326853 23189 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
W20260619 02:47:42.327131 23190 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
I20260619 02:47:42.327294 23190 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d21db929613e48019e782049d140a84a; no voters: c487c33639a44f0886a11ab6f53abd2b, f2353d3c6a7544d891f3495d1451da5d
I20260619 02:47:42.327695 23571 raft_consensus.cc:2726] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:47:42.460197 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.459365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:54245)
0619 02:47:42.459830 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:42.459971 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":223,"threads_started":1}
I20260619 02:47:42.597429 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:42.596998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:52539)
0619 02:47:42.597171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:42.597289 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:47:43.030225 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:43.029524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:56641)
0619 02:47:43.029999 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:43.030074 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":228,"threads_started":1}
I20260619 02:47:43.461807 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:43.461138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:38827)
0619 02:47:43.461537 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:43.461664 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":229,"threads_started":1}
I20260619 02:47:43.599015 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:43.598286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:60875)
0619 02:47:43.598766 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:43.598839 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":208,"threads_started":1}
I20260619 02:47:44.017871 23580 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:44.018157 23580 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:44.019022 23580 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
W20260619 02:47:44.019898 23190 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
W20260619 02:47:44.020071 23189 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
I20260619 02:47:44.020234 23189 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d21db929613e48019e782049d140a84a; no voters: c487c33639a44f0886a11ab6f53abd2b, f2353d3c6a7544d891f3495d1451da5d
I20260619 02:47:44.020519 23580 raft_consensus.cc:2726] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:47:44.031751 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:44.030972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:52253)
0619 02:47:44.031519 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:44.031596 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":335,"threads_started":1}
I20260619 02:47:44.463258 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:44.462597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:55957)
0619 02:47:44.463026 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:44.463104 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260619 02:47:44.600569 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:44.599888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:60817)
0619 02:47:44.600319 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:44.600397 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260619 02:47:45.033349 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:45.032667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:43171)
0619 02:47:45.033098 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:45.033174 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260619 02:47:45.465027 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:45.464179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:52569)
0619 02:47:45.464768 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:45.464851 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":354,"threads_started":1}
I20260619 02:47:45.602522 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:45.601839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:54071)
0619 02:47:45.602271 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:45.602352 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260619 02:47:46.034893 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:46.034253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:57763)
0619 02:47:46.034662 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:46.034736 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":228,"threads_started":1}
I20260619 02:47:46.466619 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:46.465929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:42437)
0619 02:47:46.466385 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:46.466469 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":264,"threads_started":1}
I20260619 02:47:46.604010 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:46.603372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:55141)
0619 02:47:46.603772 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:46.603846 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260619 02:47:47.036342 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:47.035683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:45733)
0619 02:47:47.036097 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:47.036173 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260619 02:47:47.133601 23591 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:47.133786 23591 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:47.134322 23591 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
W20260619 02:47:47.135283 23190 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
W20260619 02:47:47.135474 23189 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
I20260619 02:47:47.135628 23189 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d21db929613e48019e782049d140a84a; no voters: c487c33639a44f0886a11ab6f53abd2b, f2353d3c6a7544d891f3495d1451da5d
I20260619 02:47:47.135885 23591 raft_consensus.cc:2726] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:47:47.468289 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:47.467601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:32937)
0619 02:47:47.468052 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:47.468129 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260619 02:47:47.605934 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:47.604799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:60665)
0619 02:47:47.605537 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:47.605708 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":218,"threads_started":1}
I20260619 02:47:48.037990 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:48.037202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:42231)
0619 02:47:48.037732 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:48.037818 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":228,"threads_started":1}
I20260619 02:47:48.470011 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:48.469180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:46773)
0619 02:47:48.469737 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:48.469816 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":252,"threads_started":1}
I20260619 02:47:48.607549 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:48.606886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:50873)
0619 02:47:48.607302 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:48.607390 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260619 02:47:49.039463 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:49.038830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:38171)
0619 02:47:49.039219 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:49.039303 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
I20260619 02:47:49.471609 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:49.470924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:57147)
0619 02:47:49.471324 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:49.471401 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":212,"threads_started":1}
I20260619 02:47:49.608963 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:49.608331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:49149)
0619 02:47:49.608723 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:49.608799 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":214,"threads_started":1}
I20260619 02:47:50.041076 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:50.040371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:46545)
0619 02:47:50.040842 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:50.040926 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":227,"threads_started":1}
I20260619 02:47:50.473110 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:50.472434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:39143)
0619 02:47:50.472878 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:50.472954 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":212,"threads_started":1}
I20260619 02:47:50.610751 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:50.610084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:49235)
0619 02:47:50.610506 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:50.610592 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260619 02:47:51.042594 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:51.041957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:40339)
0619 02:47:51.042358 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:51.042432 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":224,"threads_started":1}
I20260619 02:47:51.474800 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:51.474119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:55153)
0619 02:47:51.474556 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:51.474637 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":237,"threads_started":1}
I20260619 02:47:51.612263 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:51.611589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:47225)
0619 02:47:51.611997 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:51.612070 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":207,"threads_started":1}
I20260619 02:47:52.044171 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:52.043426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:50957)
0619 02:47:52.043851 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:52.043929 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260619 02:47:52.108728 23607 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:52.108979 23607 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:52.109915 23607 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
W20260619 02:47:52.110760 23189 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
W20260619 02:47:52.110948 23190 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
I20260619 02:47:52.111093 23190 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d21db929613e48019e782049d140a84a; no voters: c487c33639a44f0886a11ab6f53abd2b, f2353d3c6a7544d891f3495d1451da5d
I20260619 02:47:52.111347 23607 raft_consensus.cc:2726] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:47:52.476470 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:52.475737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:40547)
0619 02:47:52.476231 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:52.476317 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260619 02:47:52.614182 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:52.613286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:58805)
0619 02:47:52.613825 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:52.613934 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
I20260619 02:47:53.045715 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:53.045001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:56241)
0619 02:47:53.045424 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:53.045498 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":227,"threads_started":1}
I20260619 02:47:53.477988 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:53.477304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:58155)
0619 02:47:53.477754 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:53.477828 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
I20260619 02:47:53.615681 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:53.615001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:50171)
0619 02:47:53.615446 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:53.615518 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":263,"threads_started":1}
I20260619 02:47:54.047173 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:54.046502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:58961)
0619 02:47:54.046911 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:54.047002 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
I20260619 02:47:54.479612 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:54.478848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:55345)
0619 02:47:54.479371 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:54.479456 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":281,"threads_started":1}
I20260619 02:47:54.617166 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:54.616473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:35937)
0619 02:47:54.616907 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:54.616986 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
I20260619 02:47:54.750813 23616 raft_consensus.cc:482] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:47:54.751034 23616 raft_consensus.cc:504] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2353d3c6a7544d891f3495d1451da5d" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39101 } } peers { permanent_uuid: "d21db929613e48019e782049d140a84a" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 33007 } } peers { permanent_uuid: "c487c33639a44f0886a11ab6f53abd2b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37165 } }
I20260619 02:47:54.751660 23616 leader_election.cc:288] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101), c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165)
W20260619 02:47:54.752693 23190 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c487c33639a44f0886a11ab6f53abd2b (127.7.246.66:37165): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
W20260619 02:47:54.752931 23189 leader_election.cc:341] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2353d3c6a7544d891f3495d1451da5d (127.7.246.67:39101): Not found: Tablet not found: df15952db4884abe9b4799843f55f8e4
I20260619 02:47:54.753082 23189 leader_election.cc:302] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d21db929613e48019e782049d140a84a; no voters: c487c33639a44f0886a11ab6f53abd2b, f2353d3c6a7544d891f3495d1451da5d
I20260619 02:47:54.753341 23616 raft_consensus.cc:2726] T df15952db4884abe9b4799843f55f8e4 P d21db929613e48019e782049d140a84a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:47:55.049026 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:55.048126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:58569)
0619 02:47:55.048654 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:55.048789 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":314,"threads_started":1}
I20260619 02:47:55.481240 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:55.480486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:40623)
0619 02:47:55.480938 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:55.481021 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260619 02:47:55.618947 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:55.618314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.65:50701)
0619 02:47:55.618701 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:55.618792 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":207,"threads_started":1}
I20260619 02:47:56.050877 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:56.050136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.66:42697)
0619 02:47:56.050613 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:56.050730 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260619 02:47:56.470623 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23138
I20260619 02:47:56.483132 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:56.482231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34285 (local address 127.7.246.67:48981)
0619 02:47:56.482652 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:56.482876 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260619 02:47:56.490828 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:56.471658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.7.246.65:33007)
0619 02:47:56.472251 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:56.472260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:56.472290 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:47:56.472359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:56.472367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:56.472647 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:56.472828 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:56.474905 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.475848 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.477167 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.477798 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.477891 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:56.484996 (+ 7105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:56.485009 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:56.485024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:56.485056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:56.487912 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.488639 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.488646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.488648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.488795 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.489175 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.489180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.489185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.489528 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:56.489896 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:56.490209 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:47:56.490487 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":253,"threads_started":1}
I20260619 02:47:56.492106 23297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:56.581489 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23138
I20260619 02:47:56.602689 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23306
I20260619 02:47:56.615445 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:56.603435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.7.246.66:37165)
0619 02:47:56.603855 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:56.603859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:56.603876 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:47:56.603928 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:56.603932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:56.603989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:56.604080 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:56.605335 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.606210 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.607358 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.607803 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.607902 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:56.610500 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:56.610505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:56.610508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:56.610547 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:56.612891 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.613469 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.613473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.613475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.613705 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.614056 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.614061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.614063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.614387 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:56.614663 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:56.614811 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:47:56.614975 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260619 02:47:56.616190 23429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:56.696028 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23306
I20260619 02:47:56.712858 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23437
I20260619 02:47:56.726303 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:56.713737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.7.246.67:39101)
0619 02:47:56.714291 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:56.714296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:56.714312 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:47:56.714364 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:56.714368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:56.714454 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:56.714563 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:56.716029 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.716794 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.718269 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:56.718651 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:56.718749 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:56.721215 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:56.721219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:56.721222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:56.721252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:56.723873 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.724492 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.724496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.724498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.724623 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:56.724943 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:56.724946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:56.724948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:56.725179 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:56.725358 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:56.725743 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 02:47:56.725940 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":302,"threads_started":1}
I20260619 02:47:56.727195 23560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:47:56.807789 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23437
2026-06-19T02:47:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24023 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:47:56 dist-test-slave-9tkz krb5kdc[23630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:47:56 dist-test-slave-9tkz krb5kdc[23630](info): set up 2 sockets
Jun 19 02:47:56 dist-test-slave-9tkz krb5kdc[23630](info): commencing operation
krb5kdc: starting...
W20260619 02:47:57.902527 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:47:58 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:47:58.137008 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:58.459564 23643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:58.459827 23643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:58.459992 23643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:58.468531 23643 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:47:58.468621 23643 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:47:58.468688 23643 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:47:58.468760 23643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:47:58.468858 23643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:58.468959 23643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:58.469022 23643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:47:58.469092 23643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:47:58.479017 23643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:58.480233 23643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:58.481623 23643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:58.493618 23648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:58.493618 23649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:58.493652 23651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:58.494266 23643 server_base.cc:1034] running on GCE node
I20260619 02:47:58.495075 23643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:58.496443 23643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:58.497691 23643 hybrid_clock.cc:648] HybridClock initialized: now 1781837278497641 us; error 49 us; skew 500 ppm
I20260619 02:47:58.498028 23643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:58 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:58.501051 23643 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:47:58.502498 23643 webserver.cc:460] Webserver started at http://127.7.246.126:37613/ using document root <none> and password file <none>
I20260619 02:47:58.503242 23643 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:58.503369 23643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:58.503681 23643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:58.506693 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "a513e4cc48894654b5a08c17d3134153"
format_stamp: "Formatted at 2026-06-19 02:47:58 on dist-test-slave-9tkz"
server_key: "baae58111429f9fdee2da33c40977323"
server_key_iv: "2ccc1df7638f6378372cc89948e7db61"
server_key_version: "encryptionkey@0"
I20260619 02:47:58.507524 23643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "a513e4cc48894654b5a08c17d3134153"
format_stamp: "Formatted at 2026-06-19 02:47:58 on dist-test-slave-9tkz"
server_key: "baae58111429f9fdee2da33c40977323"
server_key_iv: "2ccc1df7638f6378372cc89948e7db61"
server_key_version: "encryptionkey@0"
I20260619 02:47:58.512378 23643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:47:58.515736 23658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:58.517000 23643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:47:58.517191 23643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "a513e4cc48894654b5a08c17d3134153"
format_stamp: "Formatted at 2026-06-19 02:47:58 on dist-test-slave-9tkz"
server_key: "baae58111429f9fdee2da33c40977323"
server_key_iv: "2ccc1df7638f6378372cc89948e7db61"
server_key_version: "encryptionkey@0"
I20260619 02:47:58.517397 23643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:58.544257 23643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:58.548346 23643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:58.548635 23643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:58.567099 23643 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34605
I20260619 02:47:58.567121 23709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34605 every 8 connection(s)
I20260619 02:47:58.568493 23643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:47:58.572132 23710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:47:58.578248 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23643
I20260619 02:47:58.578478 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:47:58.579341 8153 external_mini_cluster.cc:1428] Setting key 9084723b3e03d3d7c40789166abd5909
I20260619 02:47:58.582859 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap starting.
I20260619 02:47:58.587330 23710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Neither blocks nor log segments found. Creating new log.
I20260619 02:47:58.588649 23710 log.cc:824] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Log is configured to *not* fsync() on all Append() calls
I20260619 02:47:58.591948 23710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: No bootstrap required, opened a new log
Jun 19 02:47:58 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:58.600042 23710 raft_consensus.cc:348] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:47:58.600419 23710 raft_consensus.cc:374] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:47:58.600546 23710 raft_consensus.cc:729] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Initialized, Role: FOLLOWER
I20260619 02:47:58.601166 23710 consensus_queue.cc:260] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:47:58.601467 23710 raft_consensus.cc:388] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:47:58.601629 23710 raft_consensus.cc:482] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:47:58.601814 23710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:47:58.604658 23710 raft_consensus.cc:504] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:47:58.605281 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:58.581479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.7.246.126:34605)
0619 02:47:58.582593 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:58.582605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:58.582665 (+ 60us) server_negotiation.cc:407] Connection header received
0619 02:47:58.585726 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:58.585773 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:58.586264 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:58.586717 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:58.588460 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:58.590525 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:58.592174 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:58.592829 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:58.593037 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:58.597513 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:58.597524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:58.597597 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:58.597644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:58.600998 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:58.601859 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:58.601866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:58.601871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:58.602087 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:58.602499 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:58.602506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:58.602508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:58.603052 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:58.603269 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:58.603737 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:47:58.604095 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":465,"threads_started":1}
I20260619 02:47:58.605230 23710 leader_election.cc:302] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a513e4cc48894654b5a08c17d3134153; no voters:
I20260619 02:47:58.605906 23710 leader_election.cc:288] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:47:58.606048 23715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:47:58.607172 23715 raft_consensus.cc:686] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 LEADER]: Becoming Leader. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Running, Role: LEADER
I20260619 02:47:58.607755 23715 consensus_queue.cc:237] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:47:58.608129 23710 sys_catalog.cc:564] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:47:58.610718 23717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a513e4cc48894654b5a08c17d3134153. Latest consensus state: current_term: 1 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:47:58.610713 23716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:47:58.611142 23717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:58.611145 23716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:47:58.618888 23722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:47:58.623449 23722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:47:58.631842 23722 catalog_manager.cc:1347] Generated new cluster ID: 814e244519a04e3bac5fbe22aa38b97e
I20260619 02:47:58.631948 23722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:47:58.646309 23722 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:47:58.648067 23722 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:47:58.659740 23722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Generated new TSK 0
I20260619 02:47:58.660524 23722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:47:58.716922 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:59.043666 23736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:59.043941 23736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:59.044042 23736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:59.051195 23736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:59.051308 23736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:59.051416 23736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:47:59.061785 23736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:59.062973 23736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:59.064286 23736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:59.077281 23741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:59.077276 23742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:59.077486 23744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:59.078131 23736 server_base.cc:1034] running on GCE node
I20260619 02:47:59.078764 23736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:59.079672 23736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:59.080886 23736 hybrid_clock.cc:648] HybridClock initialized: now 1781837279080824 us; error 59 us; skew 500 ppm
I20260619 02:47:59.081224 23736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:59 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837279, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:59.084451 23736 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:47:59.085958 23736 webserver.cc:460] Webserver started at http://127.7.246.65:43351/ using document root <none> and password file <none>
I20260619 02:47:59.086622 23736 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:59.086730 23736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:59.086977 23736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:59.090010 23736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "9948568b86b55c54a5046d98d39622ea"
server_key_iv: "7fd5e778122bd1beb41cb40a8d413459"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.091060 23736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "9948568b86b55c54a5046d98d39622ea"
server_key_iv: "7fd5e778122bd1beb41cb40a8d413459"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.096655 23736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:47:59.100270 23751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:59.101634 23736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:59.101814 23736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "9948568b86b55c54a5046d98d39622ea"
server_key_iv: "7fd5e778122bd1beb41cb40a8d413459"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.102015 23736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:59.136431 23736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:59.140678 23736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:59.140978 23736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:59.141983 23736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:59.143286 23736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:59.143393 23736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:59.143545 23736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:59.143617 23736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:59.178067 23736 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44413
I20260619 02:47:59.178098 23863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44413 every 8 connection(s)
I20260619 02:47:59.179618 23736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:47:59.189327 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23736
I20260619 02:47:59.189605 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:47:59.190433 8153 external_mini_cluster.cc:1428] Setting key b3627ca1ac9f767e8f2e47b2f9bc08c0
Jun 19 02:47:59 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837279, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:59.201191 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:59.182324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:53155 (local address 127.7.246.126:34605)
0619 02:47:59.182776 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:59.182780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:59.183655 (+ 875us) server_negotiation.cc:407] Connection header received
0619 02:47:59.185561 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:59.185568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:59.185662 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:59.185789 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:59.187353 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.188082 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.189781 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.190510 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.190674 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:59.194758 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:59.194763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:59.194767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:59.194799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:59.197697 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:59.198666 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:59.198670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:59.198672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:59.198787 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:59.199262 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:59.199265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:59.199267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:59.199520 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:59.199629 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:59.200465 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 02:47:59.200751 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":248,"threads_started":1}
I20260619 02:47:59.201884 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:59.182809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:53155)
0619 02:47:59.183492 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:59.183535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:59.185268 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:59.186033 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:59.186042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:59.186568 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:59.187172 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:59.187190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.188254 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.188258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:59.189620 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:59.189631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.190739 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.190744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:59.190945 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:59.191658 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:59.191685 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:59.194528 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:59.197836 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:59.197848 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:59.197868 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:59.198533 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:59.198891 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:59.198898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:59.198905 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:59.199159 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:59.199647 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:59.199658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:59.200097 (+ 439us) client_negotiation.cc:769] Sending connection context
0619 02:47:59.200327 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 02:47:59.200709 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260619 02:47:59.203478 23864 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:47:59.203804 23864 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:59.204661 23864 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:47:59.207075 23675 ts_manager.cc:194] Registered new tserver with Master: f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413)
I20260619 02:47:59.208645 23675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:53155
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:47:59.245784 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:47:59.559188 23870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:47:59.559461 23870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:47:59.559625 23870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:47:59.566920 23870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:47:59.567086 23870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:47:59.567270 23870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:47:59.576723 23870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:47:59.577960 23870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:47:59.579257 23870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:47:59.592079 23878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:59.592133 23875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:47:59.592172 23876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:47:59.598503 23870 server_base.cc:1034] running on GCE node
I20260619 02:47:59.599107 23870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:47:59.600040 23870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:47:59.601254 23870 hybrid_clock.cc:648] HybridClock initialized: now 1781837279601204 us; error 48 us; skew 500 ppm
I20260619 02:47:59.601590 23870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:47:59 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837279, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:47:59.604616 23870 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:47:59.606349 23870 webserver.cc:460] Webserver started at http://127.7.246.66:43787/ using document root <none> and password file <none>
I20260619 02:47:59.607156 23870 fs_manager.cc:359] Metadata directory not provided
I20260619 02:47:59.607287 23870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:47:59.607647 23870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:47:59.610545 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "a5d998a11e714902b34eb4c53f7cc31b"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "97bfced7082c7e2eb8a941d33df9b2e4"
server_key_iv: "b7236d13fb6c459c76f247e580f02184"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.611258 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d998a11e714902b34eb4c53f7cc31b"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "97bfced7082c7e2eb8a941d33df9b2e4"
server_key_iv: "b7236d13fb6c459c76f247e580f02184"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.616088 23870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:47:59.619495 23885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:59.620790 23870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:47:59.620939 23870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "a5d998a11e714902b34eb4c53f7cc31b"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "97bfced7082c7e2eb8a941d33df9b2e4"
server_key_iv: "b7236d13fb6c459c76f247e580f02184"
server_key_version: "encryptionkey@0"
I20260619 02:47:59.621125 23870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:47:59.635499 23870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:47:59.639178 23870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:47:59.639441 23870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:47:59.640291 23870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:47:59.641520 23870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:47:59.641706 23870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 02:47:59.641853 23870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:47:59.641927 23870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:47:59.674863 23870 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:46755
I20260619 02:47:59.674906 23997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:46755 every 8 connection(s)
I20260619 02:47:59.676126 23870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:47:59.678912 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 23870
I20260619 02:47:59.679123 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:47:59.679581 8153 external_mini_cluster.cc:1428] Setting key bd95e4fd2206540492836bf917d398ce
Jun 19 02:47:59 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837279, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:47:59.697407 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:59.678738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:56119 (local address 127.7.246.126:34605)
0619 02:47:59.678951 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:47:59.678957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:47:59.680195 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 02:47:59.682161 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:47:59.682167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:47:59.682255 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:47:59.682359 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:47:59.683836 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.684520 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.686134 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.686556 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.686700 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:59.690536 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:47:59.690545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:47:59.690554 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:47:59.690598 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:47:59.694124 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:59.694907 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:59.694915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:59.694921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:59.695075 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:47:59.695496 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:47:59.695504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:47:59.695510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:47:59.695914 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:47:59.696061 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:47:59.696693 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 02:47:59.696964 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 02:47:59.698091 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:47:59.679236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:56119)
0619 02:47:59.680028 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 02:47:59.680070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:47:59.681909 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:47:59.682582 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:47:59.682601 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:47:59.683118 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:47:59.683672 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:59.683688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.684666 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.684678 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:59.685978 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:47:59.685990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:47:59.686817 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:47:59.686826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:47:59.687151 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:47:59.687876 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:47:59.687910 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:47:59.690309 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:47:59.694272 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:59.694279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:59.694291 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:59.694755 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:59.695179 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:47:59.695182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:47:59.695184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:47:59.695374 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:47:59.696029 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:47:59.696034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:47:59.696328 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:47:59.696541 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 02:47:59.696934 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":346,"threads_started":1}
I20260619 02:47:59.699579 23998 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:47:59.699921 23998 heartbeater.cc:464] Registering TS with master...
I20260619 02:47:59.700871 23998 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:47:59.702620 23675 ts_manager.cc:194] Registered new tserver with Master: a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
I20260619 02:47:59.703444 23675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:56119
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:47:59.733232 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:48:00.048559 24003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:00.048887 24003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:00.049063 24003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:00.056337 24003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:00.056476 24003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:00.056658 24003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:48:00.066237 24003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:00.067539 24003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:00.068809 24003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:00.082001 24011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:00.082108 24008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:00.082126 24009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:00.083143 24003 server_base.cc:1034] running on GCE node
I20260619 02:48:00.083842 24003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:00.084899 24003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:00.086167 24003 hybrid_clock.cc:648] HybridClock initialized: now 1781837280086125 us; error 49 us; skew 500 ppm
I20260619 02:48:00.086503 24003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:00 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837280, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:00.089778 24003 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:48:00.091310 24003 webserver.cc:460] Webserver started at http://127.7.246.67:36893/ using document root <none> and password file <none>
I20260619 02:48:00.091993 24003 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:00.092144 24003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:00.092459 24003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:48:00.095374 24003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "0730afde6e2846728e1f75b905798fd3"
format_stamp: "Formatted at 2026-06-19 02:48:00 on dist-test-slave-9tkz"
server_key: "312e372fd45dc97fd905cdd683f0926c"
server_key_iv: "36fdb27ef84d3e213759f8d714de1256"
server_key_version: "encryptionkey@0"
I20260619 02:48:00.096127 24003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "0730afde6e2846728e1f75b905798fd3"
format_stamp: "Formatted at 2026-06-19 02:48:00 on dist-test-slave-9tkz"
server_key: "312e372fd45dc97fd905cdd683f0926c"
server_key_iv: "36fdb27ef84d3e213759f8d714de1256"
server_key_version: "encryptionkey@0"
I20260619 02:48:00.101063 24003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 02:48:00.104338 24018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:00.105789 24003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:00.105952 24003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "0730afde6e2846728e1f75b905798fd3"
format_stamp: "Formatted at 2026-06-19 02:48:00 on dist-test-slave-9tkz"
server_key: "312e372fd45dc97fd905cdd683f0926c"
server_key_iv: "36fdb27ef84d3e213759f8d714de1256"
server_key_version: "encryptionkey@0"
I20260619 02:48:00.106163 24003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:00.119184 24003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:00.123447 24003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:00.123680 24003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:00.124600 24003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:00.125988 24003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:00.126067 24003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:00.126152 24003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:00.126199 24003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:00.158987 24003 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:41571
I20260619 02:48:00.159034 24130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:41571 every 8 connection(s)
I20260619 02:48:00.160264 24003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:48:00.165083 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24003
I20260619 02:48:00.165302 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:48:00.165930 8153 external_mini_cluster.cc:1428] Setting key 1b041d05fe77e355f32fe7fca9dab846
Jun 19 02:48:00 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837280, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:48:00.186673 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.162993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:42691 (local address 127.7.246.126:34605)
0619 02:48:00.163544 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.163551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.164812 (+ 1261us) server_negotiation.cc:407] Connection header received
0619 02:48:00.168220 (+ 3408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.168227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.168322 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.168403 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:00.171034 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.172246 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.174193 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.174596 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.174744 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.179413 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:00.179420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:00.179424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:00.179471 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:00.182928 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.183889 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.183894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.183897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.184082 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.184636 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.184641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.184644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.185008 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:00.185150 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.186095 (+ 945us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.186318 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260619 02:48:00.188246 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.163622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:42691)
0619 02:48:00.164634 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.164686 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.167755 (+ 3069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.168945 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.168959 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.169865 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:48:00.170626 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.170644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.172434 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.172439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.174054 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.174066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.174814 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.174823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.175084 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.175913 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:48:00.175942 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:48:00.179151 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:48:00.183095 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:00.183104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:00.183120 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:00.183733 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:00.184249 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:00.184254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:00.184258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:00.184509 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:00.185240 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:48:00.185249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:48:00.185705 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.185964 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.186515 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":400,"threads_started":1}
I20260619 02:48:00.189783 24131 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:48:00.190135 24131 heartbeater.cc:464] Registering TS with master...
I20260619 02:48:00.191088 24131 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:48:00.192787 23675 ts_manager.cc:194] Registered new tserver with Master: 0730afde6e2846728e1f75b905798fd3 (127.7.246.67:41571)
I20260619 02:48:00.194149 23675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:42691
I20260619 02:48:00.197642 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:48:00.212407 23864 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:00.217024 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.203405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.7.246.126:34605)
0619 02:48:00.203607 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.203615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.203998 (+ 383us) server_negotiation.cc:407] Connection header received
0619 02:48:00.204196 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.204201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.204276 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.204378 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:00.205869 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.206666 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.208179 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.208728 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.208818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.212265 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:00.212269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:00.212272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:00.212301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:00.214906 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.215508 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.215512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.215514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.215625 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.215947 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.215950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.215952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.216232 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:00.216405 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.216650 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.216811 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:48:00.220732 23675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:48:00.227717 23675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:48:00.266157 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.239657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54824)
0619 02:48:00.256921 (+ 17264us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.256944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.257077 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.257915 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.257924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.257960 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:00.258385 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.258398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.260817 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.260823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.263277 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.263289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.265082 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.265088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.265316 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.265396 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.265541 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.265757 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17156,"thread_start_us":17028,"threads_started":1}
I20260619 02:48:00.266429 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.240521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.7.246.66:46755)
0619 02:48:00.258551 (+ 18030us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.258560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.259115 (+ 555us) server_negotiation.cc:407] Connection header received
0619 02:48:00.259301 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.259312 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.259552 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.259754 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:00.260374 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.261660 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.263758 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.265243 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.265413 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.265633 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.265907 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.266070 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17858,"thread_start_us":17718,"threads_started":1}
I20260619 02:48:00.266332 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.239955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:46755 (local address 127.0.0.1:37554)
0619 02:48:00.258972 (+ 19017us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.258999 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.259131 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.259755 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.259759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.259783 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:00.260153 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.260164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.261793 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.261800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.263563 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.263581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.265505 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.265513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.265714 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.265746 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.265835 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.265965 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18888,"mutex_wait_us":1,"thread_start_us":18750,"threads_started":1}
I20260619 02:48:00.267247 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.240360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.7.246.65:44413)
0619 02:48:00.256490 (+ 16130us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.256503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.257219 (+ 716us) server_negotiation.cc:407] Connection header received
0619 02:48:00.257328 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.257336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.257666 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.257861 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:00.258589 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.260655 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.263439 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.264847 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.265001 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.265282 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.266456 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.266636 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15922,"thread_start_us":15768,"threads_started":1}
I20260619 02:48:00.267752 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.239520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:41571 (local address 127.0.0.1:36468)
0619 02:48:00.254632 (+ 15112us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.254705 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.254949 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.260246 (+ 5297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.260252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.260277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:00.260710 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.260724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.262957 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.262963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.265162 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.265177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.267015 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.267020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.267178 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.267209 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.267272 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.267386 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14756,"mutex_wait_us":23,"thread_start_us":15666,"threads_started":1}
I20260619 02:48:00.268850 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.239682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.7.246.67:41571)
0619 02:48:00.259679 (+ 19997us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.259687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.259719 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:48:00.259812 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.259818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.260027 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.260922 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:00.261211 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.262792 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.265372 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.266789 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.268085 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.268295 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.268364 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.268512 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19838,"thread_start_us":19668,"threads_started":1}
I20260619 02:48:00.270725 23933 tablet_service.cc:1467] Processing CreateTablet for tablet 17ffff83efd54a19973e02446cbb53d1 (DEFAULT_TABLE table=test-table [id=f20e7ada39484caab7a7c599f2cc7ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:00.271118 23799 tablet_service.cc:1467] Processing CreateTablet for tablet 17ffff83efd54a19973e02446cbb53d1 (DEFAULT_TABLE table=test-table [id=f20e7ada39484caab7a7c599f2cc7ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:00.271812 23933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ffff83efd54a19973e02446cbb53d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:00.272516 23799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ffff83efd54a19973e02446cbb53d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:00.272212 24066 tablet_service.cc:1467] Processing CreateTablet for tablet 17ffff83efd54a19973e02446cbb53d1 (DEFAULT_TABLE table=test-table [id=f20e7ada39484caab7a7c599f2cc7ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:00.273526 24066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ffff83efd54a19973e02446cbb53d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:00.283336 24149 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Bootstrap starting.
I20260619 02:48:00.286505 24149 tablet_bootstrap.cc:654] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:00.286830 24150 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Bootstrap starting.
I20260619 02:48:00.288246 24149 log.cc:824] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:00.289109 24151 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Bootstrap starting.
I20260619 02:48:00.290804 24150 tablet_bootstrap.cc:654] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:00.291993 24150 log.cc:824] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:00.292266 24151 tablet_bootstrap.cc:654] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:00.293467 24151 log.cc:824] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:00.293643 24149 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: No bootstrap required, opened a new log
I20260619 02:48:00.293915 24149 ts_tablet_manager.cc:1397] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 02:48:00.294914 24150 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: No bootstrap required, opened a new log
I20260619 02:48:00.295157 24150 ts_tablet_manager.cc:1397] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 02:48:00.296056 24151 tablet_bootstrap.cc:492] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: No bootstrap required, opened a new log
I20260619 02:48:00.296299 24151 ts_tablet_manager.cc:1397] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260619 02:48:00.303483 24151 raft_consensus.cc:348] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.303949 24151 raft_consensus.cc:374] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:00.304059 24151 raft_consensus.cc:729] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0730afde6e2846728e1f75b905798fd3, State: Initialized, Role: FOLLOWER
I20260619 02:48:00.304229 24149 raft_consensus.cc:348] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.304632 24149 raft_consensus.cc:374] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:00.304730 24149 raft_consensus.cc:729] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d998a11e714902b34eb4c53f7cc31b, State: Initialized, Role: FOLLOWER
I20260619 02:48:00.304723 24151 consensus_queue.cc:260] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.305290 24149 consensus_queue.cc:260] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.305255 24150 raft_consensus.cc:348] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.305723 24150 raft_consensus.cc:374] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:00.305842 24150 raft_consensus.cc:729] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fb9cd7bd4f4e7ba08099e626848a85, State: Initialized, Role: FOLLOWER
I20260619 02:48:00.306033 24131 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:00.306798 24151 ts_tablet_manager.cc:1428] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260619 02:48:00.306608 24150 consensus_queue.cc:260] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.311479 24150 ts_tablet_manager.cc:1428] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260619 02:48:00.312443 23998 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:00.313035 24149 ts_tablet_manager.cc:1428] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260619 02:48:00.411861 24132 tablet.cc:2369] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:00.425498 24155 raft_consensus.cc:482] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:48:00.425915 24155 raft_consensus.cc:504] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.427764 24155 leader_election.cc:288] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413), a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
W20260619 02:48:00.432057 23999 tablet.cc:2369] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:48:00.432446 23865 tablet.cc:2369] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:00.436225 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.427991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.7.246.67:45907)
0619 02:48:00.428344 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.428366 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.428513 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.429185 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.429192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.429222 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:00.429895 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.429906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.431124 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.431133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.433468 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.433483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.435326 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.435333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.435539 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.435580 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.435720 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.435856 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":43}
I20260619 02:48:00.436239 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.428128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:45907 (local address 127.7.246.65:44413)
0619 02:48:00.428628 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.428637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.428658 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:48:00.428746 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.428754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.428847 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.429120 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:00.430104 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.430966 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.433702 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.435154 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.435333 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.435409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.435769 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.435925 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260619 02:48:00.437018 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.428202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:48329 (local address 127.7.246.66:46755)
0619 02:48:00.428394 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.428402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.429204 (+ 802us) server_negotiation.cc:407] Connection header received
0619 02:48:00.429466 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.429472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.429593 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.429668 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:00.430478 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.431610 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.434187 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.435620 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.435795 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.435872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.436568 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.436704 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:48:00.437100 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.428225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:46755 (local address 127.7.246.67:48329)
0619 02:48:00.429106 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.429123 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:00.429299 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:00.429875 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:00.429880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:00.429903 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:00.430309 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.430320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.431802 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.431809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.433980 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:00.434003 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.436011 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.436017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:00.436192 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.436241 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 02:48:00.436541 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 02:48:00.436670 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":4,"thread_start_us":533,"threads_started":1}
I20260619 02:48:00.437139 23819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ffff83efd54a19973e02446cbb53d1" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" is_pre_election: true
I20260619 02:48:00.437789 23819 raft_consensus.cc:2454] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 0.
I20260619 02:48:00.438596 24022 leader_election.cc:302] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0730afde6e2846728e1f75b905798fd3, f2fb9cd7bd4f4e7ba08099e626848a85; no voters:
I20260619 02:48:00.438439 23953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ffff83efd54a19973e02446cbb53d1" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d998a11e714902b34eb4c53f7cc31b" is_pre_election: true
I20260619 02:48:00.439054 24155 raft_consensus.cc:2781] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:48:00.439007 23953 raft_consensus.cc:2454] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 0.
I20260619 02:48:00.439203 24155 raft_consensus.cc:482] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:48:00.439309 24155 raft_consensus.cc:3037] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:00.441439 24155 raft_consensus.cc:504] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.442101 24155 leader_election.cc:288] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 election: Requested vote from peers f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413), a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
I20260619 02:48:00.442626 23953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ffff83efd54a19973e02446cbb53d1" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d998a11e714902b34eb4c53f7cc31b"
I20260619 02:48:00.442617 23819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ffff83efd54a19973e02446cbb53d1" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
I20260619 02:48:00.442818 23953 raft_consensus.cc:3037] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:00.442818 23819 raft_consensus.cc:3037] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:00.446012 23819 raft_consensus.cc:2454] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 1.
I20260619 02:48:00.446012 23953 raft_consensus.cc:2454] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 1.
I20260619 02:48:00.446411 24022 leader_election.cc:302] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0730afde6e2846728e1f75b905798fd3, f2fb9cd7bd4f4e7ba08099e626848a85; no voters:
I20260619 02:48:00.446696 24155 raft_consensus.cc:2781] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:48:00.447125 24155 raft_consensus.cc:686] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 0730afde6e2846728e1f75b905798fd3, State: Running, Role: LEADER
I20260619 02:48:00.447629 24155 consensus_queue.cc:237] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } }
I20260619 02:48:00.452155 23673 catalog_manager.cc:5495] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0730afde6e2846728e1f75b905798fd3 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "0730afde6e2846728e1f75b905798fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } health_report { overall_health: UNKNOWN } } }
I20260619 02:48:00.508476 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.503243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.7.246.67:41571)
0619 02:48:00.503424 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.503428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.503479 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:48:00.503673 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.503677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.503750 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.503872 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:48:00.504219 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.504899 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.506482 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.506935 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.507067 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.507472 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:48:00.507976 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:48:00.508086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.508181 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.508277 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:48:00.516598 23953 raft_consensus.cc:1261] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Refusing update from remote peer 0730afde6e2846728e1f75b905798fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:00.516637 23819 raft_consensus.cc:1261] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Refusing update from remote peer 0730afde6e2846728e1f75b905798fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:00.517616 24159 consensus_queue.cc:1035] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:00.518014 24155 consensus_queue.cc:1035] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:00.529567 24163 mvcc.cc:205] Tried to move back new op lower bound from 7298405500986707968 to 7298405500717174784. Current Snapshot: MvccSnapshot[applied={T|T < 7298405500986707968}]
I20260619 02:48:00.546545 23673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:48:00.546859 23673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:48:00.550768 23673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Sending DeleteTablet for 3 replicas of tablet 17ffff83efd54a19973e02446cbb53d1
I20260619 02:48:00.552094 23933 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ffff83efd54a19973e02446cbb53d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:00 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:37554
I20260619 02:48:00.552223 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23736
I20260619 02:48:00.552232 23799 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ffff83efd54a19973e02446cbb53d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:00 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:54824
I20260619 02:48:00.552407 24066 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ffff83efd54a19973e02446cbb53d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:00 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:36468
I20260619 02:48:00.553041 23798 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ffff83efd54a19973e02446cbb53d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:00 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:54824
I20260619 02:48:00.563580 24173 tablet_replica.cc:330] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: stopping tablet replica
I20260619 02:48:00.564304 24173 raft_consensus.cc:2229] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:48:00.565050 24173 raft_consensus.cc:2258] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 02:48:00 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:48:00.577271 24172 tablet_replica.cc:330] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: stopping tablet replica
I20260619 02:48:00.577275 24174 tablet_replica.cc:330] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: stopping tablet replica
I20260619 02:48:00.577796 24174 raft_consensus.cc:2229] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:48:00.578522 24174 raft_consensus.cc:2258] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:48:00.577831 24172 raft_consensus.cc:2229] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:48:00.581038 24174 ts_tablet_manager.cc:1905] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:00.581038 24173 ts_tablet_manager.cc:1905] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:00.586447 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.553423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.7.246.65:44413)
0619 02:48:00.553644 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.553653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.566151 (+ 12498us) server_negotiation.cc:407] Connection header received
0619 02:48:00.566367 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.566375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.566461 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.566567 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:00.568566 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.569737 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.571729 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.572337 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.572424 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.578000 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:00.578018 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:00.578036 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:00.578084 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:00.582304 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.583208 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.583222 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.583227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.583404 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.583867 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.583871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.583874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.584328 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:00.584634 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.585735 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.585975 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:48:00.588296 24172 raft_consensus.cc:2258] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:48:00.588573 23859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:00.590021 24174 ts_tablet_manager.cc:1918] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:00.590205 24174 log.cc:1196] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/17ffff83efd54a19973e02446cbb53d1
I20260619 02:48:00.591236 24174 ts_tablet_manager.cc:1939] T 17ffff83efd54a19973e02446cbb53d1 P 0730afde6e2846728e1f75b905798fd3: Deleting consensus metadata
I20260619 02:48:00.592705 24172 ts_tablet_manager.cc:1905] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:00.593114 23661 catalog_manager.cc:4841] TS 0730afde6e2846728e1f75b905798fd3 (127.7.246.67:41571): tablet 17ffff83efd54a19973e02446cbb53d1 (table test-table [id=f20e7ada39484caab7a7c599f2cc7ec4]) successfully deleted
I20260619 02:48:00.603991 24172 ts_tablet_manager.cc:1918] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:00.604174 24172 log.cc:1196] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/17ffff83efd54a19973e02446cbb53d1
I20260619 02:48:00.604704 24172 ts_tablet_manager.cc:1939] T 17ffff83efd54a19973e02446cbb53d1 P a5d998a11e714902b34eb4c53f7cc31b: Deleting consensus metadata
I20260619 02:48:00.606634 23660 catalog_manager.cc:4841] TS a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755): tablet 17ffff83efd54a19973e02446cbb53d1 (table test-table [id=f20e7ada39484caab7a7c599f2cc7ec4]) successfully deleted
I20260619 02:48:00.693042 24173 ts_tablet_manager.cc:1918] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:00.693287 24173 log.cc:1196] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/17ffff83efd54a19973e02446cbb53d1
I20260619 02:48:00.693847 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23736
I20260619 02:48:00.693881 24173 ts_tablet_manager.cc:1939] T 17ffff83efd54a19973e02446cbb53d1 P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting consensus metadata
I20260619 02:48:00.719705 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23870
W20260619 02:48:00.719784 23662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:44413 (error 108)
I20260619 02:48:00.722968 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.722466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54832)
0619 02:48:00.722675 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.722791 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 02:48:00.723377 23662 catalog_manager.cc:4584] TS f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ffff83efd54a19973e02446cbb53d1: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Jun 19 02:48:00 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:48:00.738159 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.720730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.7.246.66:46755)
0619 02:48:00.720922 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.720929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.721067 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:48:00.721423 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.721432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.721577 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.721704 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:00.723633 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.724906 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.726393 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.727038 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.727185 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.731453 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:00.731471 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:00.731492 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:00.731538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:00.735291 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.736082 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.736093 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.736097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.736252 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.736608 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.736614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.736618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.737106 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:00.737407 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.737616 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.737829 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:48:00.738955 23993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:00.768213 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.767658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54836)
0619 02:48:00.767918 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.768041 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 02:48:00.769426 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.769020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54846)
0619 02:48:00.769174 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.769289 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:48:00.776670 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.776216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54856)
0619 02:48:00.776404 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.776496 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 02:48:00.777805 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.777402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54866)
0619 02:48:00.777582 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.777668 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:48:00.818488 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.817882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54882)
0619 02:48:00.818116 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.818234 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:48:00.819747 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.819289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54886)
0619 02:48:00.819511 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.819602 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 02:48:00.833012 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23870
I20260619 02:48:00.854594 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.854102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54898)
0619 02:48:00.854331 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.854417 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:48:00.855710 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.855369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54900)
0619 02:48:00.855522 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.855598 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:48:00.856848 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24003
Jun 19 02:48:00 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:48:00.875854 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.857667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.7.246.67:41571)
0619 02:48:00.857837 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:00.857841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:00.857859 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:48:00.858007 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:00.858011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:00.858092 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:00.858167 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:00.859920 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.860776 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.862665 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:00.863105 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:00.863199 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:00.868506 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:00.868524 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:00.868544 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:00.868611 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:00.872854 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.873580 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.873592 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.873595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.873790 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:00.874178 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:00.874184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:00.874187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:00.874671 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:00.875005 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:00.875199 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:48:00.875428 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:48:00.876699 24126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:00.920769 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.920256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54916)
0619 02:48:00.920505 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.920594 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:48:00.921849 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.921454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54918)
0619 02:48:00.921665 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.921737 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 02:48:00.957276 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.956750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54932)
0619 02:48:00.957011 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.957098 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 02:48:00.958361 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:00.957993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54936)
0619 02:48:00.958139 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:00.958228 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:48:00.999933 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24003
I20260619 02:48:01.019980 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 23643
I20260619 02:48:01.021106 23705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:01.131471 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:01.130303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:54948)
0619 02:48:01.130950 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:01.131066 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:44413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260619 02:48:01.131443 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 23643
I20260619 02:48:21.152973 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--webserver_interface=127.7.246.126
--webserver_port=37613
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:48:21.481875 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:21.482239 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:21.482390 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:21.489828 24181 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:48:21.489962 24181 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:48:21.490058 24181 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:48:21.490163 24181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:48:21.490267 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:21.490374 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:21.490464 24181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:48:21.490546 24181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:48:21.501037 24181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:21.502473 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:21.503837 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:21.526530 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:21.526400 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:21.526476 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:21.527453 24181 server_base.cc:1034] running on GCE node
I20260619 02:48:21.528311 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:21.529910 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:21.531142 24181 hybrid_clock.cc:648] HybridClock initialized: now 1781837301531077 us; error 29 us; skew 500 ppm
I20260619 02:48:21.531563 24181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:21 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837301, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:21.535271 24181 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:48:21.537024 24181 webserver.cc:460] Webserver started at http://127.7.246.126:37613/ using document root <none> and password file <none>
I20260619 02:48:21.537875 24181 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:21.538028 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:21.544051 24181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 02:48:21.547163 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:21.548791 24181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:21.548988 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "a513e4cc48894654b5a08c17d3134153"
format_stamp: "Formatted at 2026-06-19 02:47:58 on dist-test-slave-9tkz"
server_key: "baae58111429f9fdee2da33c40977323"
server_key_iv: "2ccc1df7638f6378372cc89948e7db61"
server_key_version: "encryptionkey@0"
I20260619 02:48:21.550669 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:21.586329 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:21.591060 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:21.591351 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:21.610745 24181 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34605
I20260619 02:48:21.610777 24248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34605 every 8 connection(s)
I20260619 02:48:21.612319 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:48:21.617061 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24181
I20260619 02:48:21.618184 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:44413
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=43351
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:48:21.620203 24249 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:48:21.625535 24249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap starting.
I20260619 02:48:21.645717 24249 log.cc:824] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:21.658033 24249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:48:21.658583 24249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap complete.
I20260619 02:48:21.668356 24249 raft_consensus.cc:348] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:21.669158 24249 raft_consensus.cc:729] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Initialized, Role: FOLLOWER
I20260619 02:48:21.669866 24249 consensus_queue.cc:260] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:21.670120 24249 raft_consensus.cc:388] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:48:21.670236 24249 raft_consensus.cc:482] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:48:21.670423 24249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:48:21.674233 24249 raft_consensus.cc:504] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:21.674754 24249 leader_election.cc:302] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a513e4cc48894654b5a08c17d3134153; no voters:
I20260619 02:48:21.675429 24249 leader_election.cc:288] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:48:21.675581 24253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:48:21.676822 24253 raft_consensus.cc:686] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 LEADER]: Becoming Leader. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Running, Role: LEADER
I20260619 02:48:21.677474 24253 consensus_queue.cc:237] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:21.677834 24249 sys_catalog.cc:564] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:48:21.680034 24255 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a513e4cc48894654b5a08c17d3134153. Latest consensus state: current_term: 2 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:48:21.680394 24255 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:21.680146 24254 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:48:21.680552 24254 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:21.681278 24260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:48:21.693312 24260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:48:21.694993 24260 catalog_manager.cc:1259] Loaded cluster ID: 814e244519a04e3bac5fbe22aa38b97e
I20260619 02:48:21.695187 24260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:48:21.699311 24260 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:48:21.701073 24260 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Loaded TSK: 0
I20260619 02:48:21.748003 24260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Generated new TSK 1
I20260619 02:48:21.748896 24260 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:48:21.990778 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:21.991058 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:21.991200 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:21.998711 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:21.998829 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:21.998996 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:48:22.009024 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:44413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=43351
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:22.010413 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:22.011775 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:22.026619 24279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.026700 24276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.026646 24277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:22.027364 24251 server_base.cc:1034] running on GCE node
I20260619 02:48:22.027952 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:22.028954 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:22.030160 24251 hybrid_clock.cc:648] HybridClock initialized: now 1781837302030121 us; error 35 us; skew 500 ppm
I20260619 02:48:22.030517 24251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:22 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:22.033762 24251 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:48:22.035338 24251 webserver.cc:460] Webserver started at http://127.7.246.65:43351/ using document root <none> and password file <none>
I20260619 02:48:22.036341 24251 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:22.036504 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:22.042084 24251 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:48:22.044956 24286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.046756 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:22.046932 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "9948568b86b55c54a5046d98d39622ea"
server_key_iv: "7fd5e778122bd1beb41cb40a8d413459"
server_key_version: "encryptionkey@0"
I20260619 02:48:22.047931 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:22.064276 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:22.068799 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:22.069095 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:22.070160 24251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:22.071615 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:22.071755 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.071885 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:22.071975 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.106536 24251 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44413
I20260619 02:48:22.106580 24398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44413 every 8 connection(s)
I20260619 02:48:22.108006 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:48:22.114555 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24251
I20260619 02:48:22.115173 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:46755
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=43787
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:48:22 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:48:22.131251 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:22.110974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:55421 (local address 127.7.246.126:34605)
0619 02:48:22.111651 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:22.111672 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:22.112182 (+ 510us) server_negotiation.cc:407] Connection header received
0619 02:48:22.114320 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:22.114332 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:22.114679 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:22.115044 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:22.116498 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.117802 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.119430 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.120141 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.120345 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:22.123866 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:22.123874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:22.123887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:22.123923 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:22.126765 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:22.127504 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:22.127512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:22.127517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:22.127678 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:22.128103 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:22.128110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:22.128113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:22.128575 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:22.128785 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:22.129532 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 02:48:22.129945 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":274,"threads_started":1}
I20260619 02:48:22.131251 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:22.111209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:55421)
0619 02:48:22.112019 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:22.112055 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:22.113815 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:22.115183 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:22.115207 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:22.115777 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:48:22.116276 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:22.116293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.117967 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.117979 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:22.119181 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:22.119263 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.120387 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.120394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:22.120686 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:22.121400 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:48:22.121423 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:48:22.123581 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:48:22.126916 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:22.126924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:22.126938 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:22.127338 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:22.127798 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:22.127803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:22.127807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:22.127989 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:22.128716 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:48:22.128724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:48:22.129058 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:48:22.129315 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 02:48:22.129917 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":303,"threads_started":1}
I20260619 02:48:22.134140 24399 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:48:22.134511 24399 heartbeater.cc:464] Registering TS with master...
I20260619 02:48:22.135516 24399 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:48:22.137859 24214 ts_manager.cc:194] Registered new tserver with Master: f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413)
I20260619 02:48:22.139334 24214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:55421
W20260619 02:48:22.443817 24403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:22.444087 24403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:22.444187 24403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:22.451640 24403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:22.451790 24403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:22.451983 24403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:48:22.462040 24403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=43787
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:22.463407 24403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:22.464706 24403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:22.476912 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.476999 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.477034 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:22.477967 24403 server_base.cc:1034] running on GCE node
I20260619 02:48:22.478622 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:22.479593 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:22.480839 24403 hybrid_clock.cc:648] HybridClock initialized: now 1781837302480772 us; error 58 us; skew 500 ppm
I20260619 02:48:22.481151 24403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:22 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:22.484392 24403 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:48:22.485939 24403 webserver.cc:460] Webserver started at http://127.7.246.66:43787/ using document root <none> and password file <none>
I20260619 02:48:22.486606 24403 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:22.486768 24403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:22.492029 24403 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 02:48:22.494621 24418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.495865 24403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:22.496037 24403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "a5d998a11e714902b34eb4c53f7cc31b"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "97bfced7082c7e2eb8a941d33df9b2e4"
server_key_iv: "b7236d13fb6c459c76f247e580f02184"
server_key_version: "encryptionkey@0"
I20260619 02:48:22.497036 24403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:22.511785 24403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:22.516155 24403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:22.516434 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:22.517392 24403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:22.518843 24403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:22.518971 24403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.519099 24403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:22.519189 24403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.553696 24403 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:46755
I20260619 02:48:22.554324 24530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:46755 every 8 connection(s)
I20260619 02:48:22.555629 24403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:48:22.558943 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24403
I20260619 02:48:22.559737 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:41571
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=36893
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:48:22 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:48:22.577864 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:22.558262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:42275 (local address 127.7.246.126:34605)
0619 02:48:22.558442 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:22.558446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:22.559871 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 02:48:22.562873 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:22.562878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:22.562952 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:22.563043 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:22.565177 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.565935 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.568356 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.568804 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.568966 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:22.572421 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:22.572426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:22.572429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:22.572461 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:22.574862 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:22.575540 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:22.575552 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:22.575555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:22.575666 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:22.576071 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:22.576076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:22.576079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:22.576353 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:22.576507 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:22.577238 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 02:48:22.577485 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:48:22.578950 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:22.558780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:42275)
0619 02:48:22.559675 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:22.559739 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:22.562501 (+ 2762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:22.563321 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:22.563335 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:22.564143 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:48:22.564985 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:22.565014 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.566088 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.566096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:22.568208 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:22.568220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:22.569050 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:22.569058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:22.569297 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:22.570052 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:48:22.570074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:48:22.572193 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:48:22.574994 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:22.575002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:22.575018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:22.575418 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:22.575771 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:22.575775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:22.575780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:22.575964 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:22.576478 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:48:22.576485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:48:22.576904 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 02:48:22.577172 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 02:48:22.577738 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":364,"threads_started":1}
I20260619 02:48:22.580470 24531 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:48:22.580762 24531 heartbeater.cc:464] Registering TS with master...
I20260619 02:48:22.581670 24531 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:48:22.586072 24214 ts_manager.cc:194] Registered new tserver with Master: a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
I20260619 02:48:22.587229 24214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:42275
W20260619 02:48:22.911317 24534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:22.911617 24534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:22.911819 24534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:22.919394 24534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:22.919531 24534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:22.919751 24534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:48:22.929802 24534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:41571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=36893
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:22.931138 24534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:22.932550 24534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:22.946473 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.946570 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:22.946974 24539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:22.947528 24534 server_base.cc:1034] running on GCE node
I20260619 02:48:22.948122 24534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:22.949057 24534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:22.950294 24534 hybrid_clock.cc:648] HybridClock initialized: now 1781837302950238 us; error 67 us; skew 500 ppm
I20260619 02:48:22.950695 24534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:22 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:22.954054 24534 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:48:22.955675 24534 webserver.cc:460] Webserver started at http://127.7.246.67:36893/ using document root <none> and password file <none>
I20260619 02:48:22.956378 24534 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:22.956521 24534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:22.962021 24534 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:48:22.964972 24549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.966403 24534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:22.966586 24534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "0730afde6e2846728e1f75b905798fd3"
format_stamp: "Formatted at 2026-06-19 02:48:00 on dist-test-slave-9tkz"
server_key: "312e372fd45dc97fd905cdd683f0926c"
server_key_iv: "36fdb27ef84d3e213759f8d714de1256"
server_key_version: "encryptionkey@0"
I20260619 02:48:22.967609 24534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:22.984391 24534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:22.988397 24534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:22.988675 24534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:22.989707 24534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:22.991003 24534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:22.991120 24534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:22.991245 24534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:22.991326 24534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:23.025177 24534 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:41571
I20260619 02:48:23.025265 24661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:41571 every 8 connection(s)
I20260619 02:48:23.026710 24534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:48:23.036288 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24534
Jun 19 02:48:23 dist-test-slave-9tkz krb5kdc[23630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837302, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:48:23.047776 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.029192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:60737 (local address 127.7.246.126:34605)
0619 02:48:23.029362 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.029366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.030430 (+ 1064us) server_negotiation.cc:407] Connection header received
0619 02:48:23.032224 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.032230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.032320 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.032426 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.033932 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.034671 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.036461 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.036948 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.037100 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.041216 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.041222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.041226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.041263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.044363 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.045044 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.045049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.045053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.045200 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.045774 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.045780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.045784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.046165 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.046341 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.047059 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.047307 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:48:23.048728 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.029601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:60737)
0619 02:48:23.030268 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.030306 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.031968 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.032649 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.032657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.033128 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:48:23.033764 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.033789 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.034855 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.034859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.036261 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.036276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.037195 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.037203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.037544 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.038369 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:48:23.038403 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:48:23.041025 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:48:23.044484 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:23.044489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:23.044501 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:23.044907 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:23.045357 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:48:23.045361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:48:23.045364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:48:23.045644 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:48:23.046300 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:48:23.046305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:48:23.046656 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.046939 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.047515 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":259,"threads_started":1}
I20260619 02:48:23.050231 24662 heartbeater.cc:347] Connected to a master server at 127.7.246.126:34605
I20260619 02:48:23.050530 24662 heartbeater.cc:464] Registering TS with master...
I20260619 02:48:23.051491 24662 heartbeater.cc:511] Master 127.7.246.126:34605 requested a full tablet report, sending...
I20260619 02:48:23.052691 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.037185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.7.246.126:34605)
0619 02:48:23.037815 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.037821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.038116 (+ 295us) server_negotiation.cc:407] Connection header received
0619 02:48:23.038358 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.038362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.038428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.038512 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.040487 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.041250 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.042952 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.043680 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.043824 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.046805 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.046809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.046812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.046837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.049610 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.050382 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.050386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.050389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.050515 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.051009 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.051013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.051016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.051386 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.051658 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.052075 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.052287 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":289,"threads_started":1}
I20260619 02:48:23.053223 24214 ts_manager.cc:194] Registered new tserver with Master: 0730afde6e2846728e1f75b905798fd3 (127.7.246.67:41571)
I20260619 02:48:23.054312 24214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:60737
I20260619 02:48:23.055533 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:48:23.061983 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.056607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.7.246.126:34605)
0619 02:48:23.056818 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.056827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.057143 (+ 316us) server_negotiation.cc:407] Connection header received
0619 02:48:23.057343 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.057349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.057493 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.057710 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:48:23.058071 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.058849 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.060247 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.060674 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.060770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.061183 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:48:23.061362 (+ 179us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 02:48:23.061725 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60366: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:48:23.062104 24665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60366: token expired
W20260619 02:48:23.062604 24134 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.7.246.126:34605)
I20260619 02:48:23.075488 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.063096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.7.246.126:34605)
0619 02:48:23.063283 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.063291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.063309 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:48:23.063352 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.063356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.063413 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.063502 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.064909 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.065679 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.067215 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.067600 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.067722 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.070128 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.070132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.070135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.070163 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.072833 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.073442 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.073447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.073450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.073627 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.074049 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.074054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.074057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.074352 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.074608 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.074911 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.075109 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:48:23.079881 24213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:48:23.089306 24213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:48:23.117070 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.102552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:46755 (local address 127.0.0.1:44442)
0619 02:48:23.104339 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.104356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.104437 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.108868 (+ 4431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.108875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.108903 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.109292 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.109310 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.111492 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.111498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.114192 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.114204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.116096 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.116112 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.116322 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.116385 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.116488 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.116680 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"mutex_wait_us":2,"thread_start_us":430,"threads_started":1}
I20260619 02:48:23.117421 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.103236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.7.246.66:46755)
0619 02:48:23.106550 (+ 3314us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.106557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.106584 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:48:23.106722 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.106730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.106983 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.107181 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.109580 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.111317 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.114340 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.115852 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.116561 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.116765 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.116838 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.117001 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3173,"thread_start_us":3084,"threads_started":1}
I20260619 02:48:23.118245 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.103658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.7.246.67:41571)
0619 02:48:23.106173 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.106183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.108282 (+ 2099us) server_negotiation.cc:407] Connection header received
0619 02:48:23.108447 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.108456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.108754 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.108946 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.110070 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.112028 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.115094 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.116801 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.116933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.117155 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.117718 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.117901 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2342,"thread_start_us":2174,"threads_started":1}
I20260619 02:48:23.118990 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.102984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:41571 (local address 127.0.0.1:34612)
0619 02:48:23.108127 (+ 5143us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.108146 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.108287 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.109295 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.109303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.109339 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.109836 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.109852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.112227 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.112234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.114932 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.114945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.117288 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.117294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.117474 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.117529 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.118443 (+ 914us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.118594 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5037,"mutex_wait_us":27,"thread_start_us":1139,"threads_started":1}
I20260619 02:48:23.121243 24466 tablet_service.cc:1467] Processing CreateTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a (DEFAULT_TABLE table=test-table [id=69b3dacbc6d344c2a21dca9b893e2062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:23.122071 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.102398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.0.0.1:44374)
0619 02:48:23.103348 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.103382 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.103542 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.115357 (+ 11815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.115362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.115391 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.115763 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.115776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.117655 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.117662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.119533 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.119545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.121312 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.121319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.121488 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.121524 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.121628 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.121748 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":5490,"threads_started":1}
I20260619 02:48:23.122792 24466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad334b5a7e794a4a86c604f0edc3c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:23.123157 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.102587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.7.246.65:44413)
0619 02:48:23.114819 (+ 12232us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.114827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.114854 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:48:23.114933 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.114940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.115152 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.115351 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.115972 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.117464 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.119699 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.121057 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.122396 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.122630 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.122695 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.122863 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12059,"thread_start_us":11909,"threads_started":1}
I20260619 02:48:23.122890 24597 tablet_service.cc:1467] Processing CreateTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a (DEFAULT_TABLE table=test-table [id=69b3dacbc6d344c2a21dca9b893e2062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:23.124153 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad334b5a7e794a4a86c604f0edc3c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:23.126617 24334 tablet_service.cc:1467] Processing CreateTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a (DEFAULT_TABLE table=test-table [id=69b3dacbc6d344c2a21dca9b893e2062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:48:23.127873 24334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad334b5a7e794a4a86c604f0edc3c78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:48:23.135421 24678 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Bootstrap starting.
I20260619 02:48:23.138491 24678 tablet_bootstrap.cc:654] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:23.139187 24679 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Bootstrap starting.
I20260619 02:48:23.140895 24678 log.cc:824] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:23.142700 24681 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Bootstrap starting.
I20260619 02:48:23.143365 24679 tablet_bootstrap.cc:654] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:23.144014 24678 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: No bootstrap required, opened a new log
I20260619 02:48:23.144214 24678 ts_tablet_manager.cc:1397] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:48:23.144603 24679 log.cc:824] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:23.145565 24399 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:23.146136 24681 tablet_bootstrap.cc:654] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Neither blocks nor log segments found. Creating new log.
I20260619 02:48:23.147293 24681 log.cc:824] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:23.147516 24679 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: No bootstrap required, opened a new log
I20260619 02:48:23.147751 24679 ts_tablet_manager.cc:1397] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 02:48:23.152705 24681 tablet_bootstrap.cc:492] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: No bootstrap required, opened a new log
I20260619 02:48:23.152984 24681 ts_tablet_manager.cc:1397] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 02:48:23.154724 24678 raft_consensus.cc:348] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.155189 24678 raft_consensus.cc:374] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:23.155288 24678 raft_consensus.cc:729] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d998a11e714902b34eb4c53f7cc31b, State: Initialized, Role: FOLLOWER
I20260619 02:48:23.156029 24678 consensus_queue.cc:260] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.157150 24531 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:23.157860 24678 ts_tablet_manager.cc:1428] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 02:48:23.158674 24679 raft_consensus.cc:348] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.159108 24679 raft_consensus.cc:374] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:23.159245 24679 raft_consensus.cc:729] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0730afde6e2846728e1f75b905798fd3, State: Initialized, Role: FOLLOWER
I20260619 02:48:23.159915 24679 consensus_queue.cc:260] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.161711 24679 ts_tablet_manager.cc:1428] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 02:48:23.162212 24662 heartbeater.cc:503] Master 127.7.246.126:34605 was elected leader, sending a full tablet report...
I20260619 02:48:23.162397 24681 raft_consensus.cc:348] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.162746 24681 raft_consensus.cc:374] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:48:23.162854 24681 raft_consensus.cc:729] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fb9cd7bd4f4e7ba08099e626848a85, State: Initialized, Role: FOLLOWER
I20260619 02:48:23.163435 24681 consensus_queue.cc:260] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.165102 24681 ts_tablet_manager.cc:1428] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260619 02:48:23.227388 24685 raft_consensus.cc:482] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:48:23.227689 24685 raft_consensus.cc:504] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.229430 24685 leader_election.cc:288] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413), a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
I20260619 02:48:23.231343 24686 raft_consensus.cc:482] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:48:23.231678 24686 raft_consensus.cc:504] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.233506 24686 leader_election.cc:288] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755), 0730afde6e2846728e1f75b905798fd3 (127.7.246.67:41571)
I20260619 02:48:23.237624 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.229771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:44413 (local address 127.7.246.67:37029)
0619 02:48:23.230087 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.230107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.230239 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.230575 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.230580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.230627 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.230831 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.230839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.232168 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.232176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.234926 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.234940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.236754 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.236761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.236957 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.237005 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.237088 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.237236 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":979}
I20260619 02:48:23.237861 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.229946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:58377 (local address 127.7.246.66:46755)
0619 02:48:23.230148 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.230156 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.230719 (+ 563us) server_negotiation.cc:407] Connection header received
0619 02:48:23.230924 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.230928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.230982 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.231085 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.231708 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.232755 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.235367 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.236612 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.236742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.236812 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.237433 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.237595 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:48:23.238489 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.229880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:37029 (local address 127.7.246.65:44413)
0619 02:48:23.230096 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.230104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.230332 (+ 228us) server_negotiation.cc:407] Connection header received
0619 02:48:23.230397 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.230400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.230456 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.230550 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.230970 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.232062 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.235123 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.236614 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.236759 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.236818 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.237898 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.238186 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":69}
I20260619 02:48:23.238591 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.229879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:46755 (local address 127.7.246.67:58377)
0619 02:48:23.230580 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.230614 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.230782 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.231125 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.231131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.231157 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.231500 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.231513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.232905 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.232910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.235204 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.235218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.237011 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.237017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.237209 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.237251 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.238073 (+ 822us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.238200 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":26,"thread_start_us":381,"threads_started":1}
I20260619 02:48:23.239321 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" is_pre_election: true
I20260619 02:48:23.239555 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d998a11e714902b34eb4c53f7cc31b" is_pre_election: true
I20260619 02:48:23.239789 24354 raft_consensus.cc:2454] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 0.
I20260619 02:48:23.240085 24486 raft_consensus.cc:2454] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 0.
I20260619 02:48:23.240562 24553 leader_election.cc:302] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0730afde6e2846728e1f75b905798fd3, f2fb9cd7bd4f4e7ba08099e626848a85; no voters:
I20260619 02:48:23.241284 24685 raft_consensus.cc:2781] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:48:23.241430 24685 raft_consensus.cc:482] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:48:23.241532 24685 raft_consensus.cc:3037] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:23.243142 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.233883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:47821 (local address 127.7.246.67:41571)
0619 02:48:23.234105 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.234140 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.235280 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 02:48:23.235597 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.235605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.235689 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.235759 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.237485 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.238584 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.241211 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.242232 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.242409 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.242480 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.242752 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.242873 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:48:23.243394 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.233885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:41571 (local address 127.7.246.65:47821)
0619 02:48:23.235167 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.235189 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.235405 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.236015 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.236022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.236056 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.237315 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.237331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.238775 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.238783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.241065 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.241076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.242434 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.242440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.242608 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.242631 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.242732 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.242851 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"mutex_wait_us":800,"thread_start_us":2911,"threads_started":1}
I20260619 02:48:23.244201 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0730afde6e2846728e1f75b905798fd3" is_pre_election: true
I20260619 02:48:23.244792 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.234474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:46755 (local address 127.7.246.65:54507)
0619 02:48:23.236688 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:23.236704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:48:23.236877 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:48:23.237397 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:48:23.237401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:48:23.237445 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:48:23.238167 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.238180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.240066 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.240075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.242243 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:48:23.242256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.244082 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.244088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:48:23.244253 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.244283 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:48:23.244363 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:48:23.244484 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2101,"thread_start_us":1937,"threads_started":1}
I20260619 02:48:23.244856 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.233874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:54507 (local address 127.7.246.66:46755)
0619 02:48:23.234618 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.234625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.236770 (+ 2145us) server_negotiation.cc:407] Connection header received
0619 02:48:23.237015 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.237020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.237088 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.237149 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:48:23.238384 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.239556 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.242404 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.243664 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.243827 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.243897 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.244439 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.244566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":750,"threads_started":1}
I20260619 02:48:23.245185 24685 raft_consensus.cc:504] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.245376 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d998a11e714902b34eb4c53f7cc31b" is_pre_election: true
I20260619 02:48:23.245637 24486 raft_consensus.cc:2454] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fb9cd7bd4f4e7ba08099e626848a85 in term 0.
I20260619 02:48:23.245723 24617 raft_consensus.cc:2379] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fb9cd7bd4f4e7ba08099e626848a85 in current term 1: Already voted for candidate 0730afde6e2846728e1f75b905798fd3 in this term.
I20260619 02:48:23.246001 24685 leader_election.cc:288] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 election: Requested vote from peers f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413), a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755)
I20260619 02:48:23.246356 24486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d998a11e714902b34eb4c53f7cc31b"
I20260619 02:48:23.246407 24354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad334b5a7e794a4a86c604f0edc3c78a" candidate_uuid: "0730afde6e2846728e1f75b905798fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
I20260619 02:48:23.246537 24486 raft_consensus.cc:3037] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:23.246629 24354 raft_consensus.cc:3037] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:48:23.247416 24288 leader_election.cc:302] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d998a11e714902b34eb4c53f7cc31b, f2fb9cd7bd4f4e7ba08099e626848a85; no voters:
I20260619 02:48:23.249150 24486 raft_consensus.cc:2454] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 1.
I20260619 02:48:23.249454 24354 raft_consensus.cc:2454] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0730afde6e2846728e1f75b905798fd3 in term 1.
I20260619 02:48:23.249476 24551 leader_election.cc:302] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0730afde6e2846728e1f75b905798fd3, a5d998a11e714902b34eb4c53f7cc31b; no voters:
I20260619 02:48:23.249755 24686 raft_consensus.cc:2741] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 02:48:23.249817 24685 raft_consensus.cc:2781] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:48:23.250276 24685 raft_consensus.cc:686] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 0730afde6e2846728e1f75b905798fd3, State: Running, Role: LEADER
I20260619 02:48:23.250852 24685 consensus_queue.cc:237] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } }
I20260619 02:48:23.255132 24214 catalog_manager.cc:5495] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0730afde6e2846728e1f75b905798fd3 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "0730afde6e2846728e1f75b905798fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0730afde6e2846728e1f75b905798fd3" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 41571 } health_report { overall_health: HEALTHY } } }
W20260619 02:48:23.278247 24663 tablet.cc:2369] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:23.304822 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.299635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.7.246.67:41571)
0619 02:48:23.299817 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.299822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.299861 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:48:23.300046 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.300050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.300144 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.300298 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:48:23.300714 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.301614 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.302954 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.303311 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.303395 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.303824 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:48:23.304271 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:48:23.304389 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.304451 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.304509 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:48:23.307674 24532 tablet.cc:2369] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:48:23.313037 24486 raft_consensus.cc:1261] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Refusing update from remote peer 0730afde6e2846728e1f75b905798fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:23.313020 24354 raft_consensus.cc:1261] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Refusing update from remote peer 0730afde6e2846728e1f75b905798fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:48:23.313949 24691 consensus_queue.cc:1035] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fb9cd7bd4f4e7ba08099e626848a85" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 44413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:23.314319 24685 consensus_queue.cc:1035] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d998a11e714902b34eb4c53f7cc31b" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:48:23.329123 24694 mvcc.cc:205] Tried to move back new op lower bound from 7298405594360356864 to 7298405594119774208. Current Snapshot: MvccSnapshot[applied={T|T < 7298405594360356864}]
I20260619 02:48:23.338250 24214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:48:23.338537 24214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:48:23.341650 24214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Sending DeleteTablet for 3 replicas of tablet ad334b5a7e794a4a86c604f0edc3c78a
I20260619 02:48:23.342681 24334 tablet_service.cc:1514] Processing DeleteTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:44374
I20260619 02:48:23.342698 24466 tablet_service.cc:1514] Processing DeleteTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:44442
I20260619 02:48:23.343336 24597 tablet_service.cc:1514] Processing DeleteTablet for tablet ad334b5a7e794a4a86c604f0edc3c78a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:48:23 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:34612
I20260619 02:48:23.343573 24703 tablet_replica.cc:330] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: stopping tablet replica
I20260619 02:48:23.343894 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24251
I20260619 02:48:23.344055 24703 raft_consensus.cc:2229] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:48:23.344579 24703 raft_consensus.cc:2258] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:48:23.346783 24703 ts_tablet_manager.cc:1905] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:23.354113 24703 ts_tablet_manager.cc:1918] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:23.354342 24703 log.cc:1196] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/ad334b5a7e794a4a86c604f0edc3c78a
I20260619 02:48:23.354964 24703 ts_tablet_manager.cc:1939] T ad334b5a7e794a4a86c604f0edc3c78a P f2fb9cd7bd4f4e7ba08099e626848a85: Deleting consensus metadata
I20260619 02:48:23.356891 24201 catalog_manager.cc:4841] TS f2fb9cd7bd4f4e7ba08099e626848a85 (127.7.246.65:44413): tablet ad334b5a7e794a4a86c604f0edc3c78a (table test-table [id=69b3dacbc6d344c2a21dca9b893e2062]) successfully deleted
I20260619 02:48:23.359390 24705 tablet_replica.cc:330] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: stopping tablet replica
I20260619 02:48:23.360376 24704 tablet_replica.cc:330] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: stopping tablet replica
I20260619 02:48:23.360791 24704 raft_consensus.cc:2229] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:48:23.361287 24704 raft_consensus.cc:2258] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:48:23.362018 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.345239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.7.246.65:44413)
0619 02:48:23.345382 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.345389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.345410 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:48:23.345481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.345488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.345587 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.345681 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.347726 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.348866 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.350818 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.351507 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.351628 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.355498 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.355523 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.355541 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.355576 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.358836 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.359648 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.359659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.359663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.359831 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.360308 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.360315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.360318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.360729 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.361076 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.361383 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.361636 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:48:23.361984 24705 raft_consensus.cc:2229] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:48:23.362641 24394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:23.362648 24705 raft_consensus.cc:2258] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:48:23.367348 24704 ts_tablet_manager.cc:1905] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:23.367806 24705 ts_tablet_manager.cc:1905] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:48:23.374220 24705 ts_tablet_manager.cc:1918] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:23.374390 24705 log.cc:1196] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/ad334b5a7e794a4a86c604f0edc3c78a
I20260619 02:48:23.374845 24705 ts_tablet_manager.cc:1939] T ad334b5a7e794a4a86c604f0edc3c78a P 0730afde6e2846728e1f75b905798fd3: Deleting consensus metadata
I20260619 02:48:23.376269 24200 catalog_manager.cc:4841] TS 0730afde6e2846728e1f75b905798fd3 (127.7.246.67:41571): tablet ad334b5a7e794a4a86c604f0edc3c78a (table test-table [id=69b3dacbc6d344c2a21dca9b893e2062]) successfully deleted
I20260619 02:48:23.380340 24704 ts_tablet_manager.cc:1918] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:48:23.380504 24704 log.cc:1196] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/ad334b5a7e794a4a86c604f0edc3c78a
I20260619 02:48:23.380936 24704 ts_tablet_manager.cc:1939] T ad334b5a7e794a4a86c604f0edc3c78a P a5d998a11e714902b34eb4c53f7cc31b: Deleting consensus metadata
I20260619 02:48:23.382400 24199 catalog_manager.cc:4841] TS a5d998a11e714902b34eb4c53f7cc31b (127.7.246.66:46755): tablet ad334b5a7e794a4a86c604f0edc3c78a (table test-table [id=69b3dacbc6d344c2a21dca9b893e2062]) successfully deleted
I20260619 02:48:23.468690 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24251
I20260619 02:48:23.487139 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24403
I20260619 02:48:23.503146 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.487931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.7.246.66:46755)
0619 02:48:23.488111 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.488116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.488193 (+ 77us) server_negotiation.cc:407] Connection header received
0619 02:48:23.488368 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.488372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.488433 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.488514 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.490438 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.491105 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.493093 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.493496 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.493677 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.497078 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.497091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.497105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.497137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.500399 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.501292 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.501298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.501300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.501423 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.501831 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.501834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.501836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.502176 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.502401 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.502720 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.502901 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:48:23.503756 24526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:23.595304 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24403
I20260619 02:48:23.613312 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24534
I20260619 02:48:23.631928 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:23.618313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.7.246.67:41571)
0619 02:48:23.618493 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:48:23.618501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:48:23.618520 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:48:23.618759 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:48:23.618763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:48:23.618826 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:48:23.618952 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:48:23.620378 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.621111 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.622541 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:48:23.623074 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:48:23.623152 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:48:23.625773 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:48:23.625788 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:48:23.625809 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:48:23.625854 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:48:23.628982 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.629804 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.629813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.629816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.629953 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:48:23.630357 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:48:23.630363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:48:23.630365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:48:23.630743 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:48:23.631032 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 02:48:23.631345 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:48:23.631559 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:48:23.632637 24657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:23.752645 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24534
I20260619 02:48:23.772387 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24181
I20260619 02:48:23.773609 24244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:48:23.885234 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24181
I20260619 02:48:43.907308 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--webserver_interface=127.7.246.126
--webserver_port=37613
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:48:44.243708 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:44.243994 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:44.244167 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:44.251482 24710 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:48:44.251569 24710 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:48:44.251657 24710 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:48:44.251748 24710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:48:44.251833 24710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:44.251948 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:44.252022 24710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:48:44.252106 24710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:48:44.261698 24710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:44.263031 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:44.264555 24710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:44.276072 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:44.276063 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:44.276408 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:44.276901 24710 server_base.cc:1034] running on GCE node
I20260619 02:48:44.277773 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:44.279115 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:44.280354 24710 hybrid_clock.cc:648] HybridClock initialized: now 1781837324280289 us; error 38 us; skew 500 ppm
I20260619 02:48:44.280682 24710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:44 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837324, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:44.286834 24710 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:48:44.288529 24710 webserver.cc:460] Webserver started at http://127.7.246.126:37613/ using document root <none> and password file <none>
I20260619 02:48:44.289276 24710 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:44.289417 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:44.295087 24710 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 02:48:44.298050 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:44.299422 24710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:48:44.299620 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "a513e4cc48894654b5a08c17d3134153"
format_stamp: "Formatted at 2026-06-19 02:47:58 on dist-test-slave-9tkz"
server_key: "baae58111429f9fdee2da33c40977323"
server_key_iv: "2ccc1df7638f6378372cc89948e7db61"
server_key_version: "encryptionkey@0"
I20260619 02:48:44.300750 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:44.322467 24710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:44.326908 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:44.327230 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:44.345914 24710 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:34605
I20260619 02:48:44.345957 24776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:34605 every 8 connection(s)
I20260619 02:48:44.347427 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:48:44.350893 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24710
I20260619 02:48:44.351889 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:44413
--local_ip_for_outbound_sockets=127.7.246.65
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=43351
--webserver_interface=127.7.246.65
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:48:44.355921 24777 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:48:44.361043 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap starting.
I20260619 02:48:44.380923 24777 log.cc:824] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Log is configured to *not* fsync() on all Append() calls
I20260619 02:48:44.396435 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:48:44.397004 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Bootstrap complete.
I20260619 02:48:44.406437 24777 raft_consensus.cc:348] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:44.407297 24777 raft_consensus.cc:729] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Initialized, Role: FOLLOWER
I20260619 02:48:44.407884 24777 consensus_queue.cc:260] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:44.408102 24777 raft_consensus.cc:388] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:48:44.408210 24777 raft_consensus.cc:482] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:48:44.408393 24777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:48:44.411962 24777 raft_consensus.cc:504] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:44.412411 24777 leader_election.cc:302] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a513e4cc48894654b5a08c17d3134153; no voters:
I20260619 02:48:44.413084 24777 leader_election.cc:288] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:48:44.413275 24781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:48:44.414414 24781 raft_consensus.cc:686] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [term 3 LEADER]: Becoming Leader. State: Replica: a513e4cc48894654b5a08c17d3134153, State: Running, Role: LEADER
I20260619 02:48:44.415074 24777 sys_catalog.cc:564] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:48:44.415030 24781 consensus_queue.cc:237] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } }
I20260619 02:48:44.417614 24783 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a513e4cc48894654b5a08c17d3134153. Latest consensus state: current_term: 3 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:48:44.417966 24783 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:44.418535 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a513e4cc48894654b5a08c17d3134153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a513e4cc48894654b5a08c17d3134153" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 34605 } } }
I20260619 02:48:44.418893 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153 [sys.catalog]: This master's current role is: LEADER
I20260619 02:48:44.426162 24787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:48:44.434545 24787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:48:44.436043 24787 catalog_manager.cc:1259] Loaded cluster ID: 814e244519a04e3bac5fbe22aa38b97e
I20260619 02:48:44.436173 24787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:48:44.440200 24787 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:48:44.441847 24787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Loaded TSK: 0
I20260619 02:48:44.442022 24787 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a513e4cc48894654b5a08c17d3134153: Loaded TSK: 1
F20260619 02:48:44.443149 24787 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f17c006a9ad google::LogMessage::Fail() at ??:0
@ 0x7f17c006d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f17c006a4de google::LogMessage::Flush() at ??:0
@ 0x7f17c006ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f17c3dce7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f17c3dbd73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f17d5a991c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f17d5a98783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f17d5b242a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f17d5b240c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f17d5b6e5e9 std::function<>::operator()() at ??:0
@ 0x7f17d5a9ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f17d5a9ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f17d5b22610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f17d5b22439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17c2cb6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f17c2cb991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f17c2cb9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17c2c94eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f17c5f6e6db start_thread at ??:0
@ 0x7f17bef0461f clone at ??:0
W20260619 02:48:44.750651 24779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:44.750982 24779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:44.751147 24779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:44.758320 24779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:44.758488 24779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:44.758675 24779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:48:44.768221 24779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:44413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=43351
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:44.769503 24779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:44.770824 24779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:44.783540 24825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:44.783571 24824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:44.784103 24827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:44.784374 24779 server_base.cc:1034] running on GCE node
I20260619 02:48:44.785053 24779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:44.785986 24779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:44.787186 24779 hybrid_clock.cc:648] HybridClock initialized: now 1781837324787149 us; error 32 us; skew 500 ppm
I20260619 02:48:44.787489 24779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:44 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837324, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:44.790597 24779 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:48:44.792034 24779 webserver.cc:460] Webserver started at http://127.7.246.65:43351/ using document root <none> and password file <none>
I20260619 02:48:44.792686 24779 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:44.792819 24779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:44.797933 24779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:48:44.800514 24834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:44.801839 24779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:48:44.802000 24779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "f2fb9cd7bd4f4e7ba08099e626848a85"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "9948568b86b55c54a5046d98d39622ea"
server_key_iv: "7fd5e778122bd1beb41cb40a8d413459"
server_key_version: "encryptionkey@0"
I20260619 02:48:44.802925 24779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:44.818080 24779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:44.822132 24779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:44.822392 24779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:44.823289 24779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:44.824541 24779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:44.824664 24779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:44.824787 24779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:44.824872 24779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:44.858050 24779 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:44413
I20260619 02:48:44.858130 24946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:44413 every 8 connection(s)
I20260619 02:48:44.859519 24779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:48:44.863492 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24779
I20260619 02:48:44.864338 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:46755
--local_ip_for_outbound_sockets=127.7.246.66
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=43787
--webserver_interface=127.7.246.66
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:48:44.865025 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:44.862513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:59269)
0619 02:48:44.863488 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:44.863773 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":410,"threads_started":1}
W20260619 02:48:44.865857 24947 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34605 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34605: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
I20260619 02:48:44.866807 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:44.866459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:60373)
0619 02:48:44.866609 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:44.866697 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:48:44.867813 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:44.867428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:58939)
0619 02:48:44.867587 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:44.867673 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:48:44.868067 24947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:48:45.185117 24950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:45.185458 24950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:45.185644 24950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:45.192843 24950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:45.192948 24950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:45.193061 24950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:48:45.202667 24950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=43787
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:45.203776 24950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:45.205039 24950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:45.217875 24955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:45.217851 24958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:45.217922 24956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:45.219122 24950 server_base.cc:1034] running on GCE node
I20260619 02:48:45.219740 24950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:45.220628 24950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:45.221846 24950 hybrid_clock.cc:648] HybridClock initialized: now 1781837325221788 us; error 56 us; skew 500 ppm
I20260619 02:48:45.222187 24950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:45 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837325, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:45.225243 24950 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:48:45.227008 24950 webserver.cc:460] Webserver started at http://127.7.246.66:43787/ using document root <none> and password file <none>
I20260619 02:48:45.227681 24950 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:45.227820 24950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:45.233014 24950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:48:45.235770 24965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.237155 24950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:48:45.237332 24950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "a5d998a11e714902b34eb4c53f7cc31b"
format_stamp: "Formatted at 2026-06-19 02:47:59 on dist-test-slave-9tkz"
server_key: "97bfced7082c7e2eb8a941d33df9b2e4"
server_key_iv: "b7236d13fb6c459c76f247e580f02184"
server_key_version: "encryptionkey@0"
I20260619 02:48:45.238358 24950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:45.259778 24950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:45.263585 24950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:45.263878 24950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:45.264815 24950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:45.266170 24950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:45.266285 24950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.266428 24950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:45.266525 24950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.299393 24950 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:46755
I20260619 02:48:45.299438 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:46755 every 8 connection(s)
I20260619 02:48:45.301219 24950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:48:45.306434 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.304338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:43359)
0619 02:48:45.305076 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.305311 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
W20260619 02:48:45.307088 25078 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34605 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34605: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
I20260619 02:48:45.307824 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 24950
I20260619 02:48:45.308039 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.307703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:40627)
0619 02:48:45.307846 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.307908 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:48:45.308620 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:41571
--local_ip_for_outbound_sockets=127.7.246.67
--tserver_master_addrs=127.7.246.126:34605
--webserver_port=36893
--webserver_interface=127.7.246.67
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:48:45.308835 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.308499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:49391)
0619 02:48:45.308626 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.308701 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:48:45.309055 25078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:48:45.653913 25081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:48:45.654197 25081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:48:45.654362 25081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:48:45.661599 25081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:48:45.661724 25081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:48:45.661893 25081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:48:45.671702 25081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:41571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=36893
--tserver_master_addrs=127.7.246.126:34605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:48:45.672930 25081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:48:45.674283 25081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:48:45.687110 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:45.687216 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:48:45.687183 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:48:45.688124 25081 server_base.cc:1034] running on GCE node
I20260619 02:48:45.688731 25081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:48:45.689628 25081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:48:45.690824 25081 hybrid_clock.cc:648] HybridClock initialized: now 1781837325690768 us; error 56 us; skew 500 ppm
I20260619 02:48:45.691092 25081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:48:45 dist-test-slave-9tkz krb5kdc[23630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837325, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:48:45.694031 25081 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:48:45.695432 25081 webserver.cc:460] Webserver started at http://127.7.246.67:36893/ using document root <none> and password file <none>
I20260619 02:48:45.696084 25081 fs_manager.cc:359] Metadata directory not provided
I20260619 02:48:45.696206 25081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:48:45.701419 25081 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:48:45.703984 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.705345 25081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:48:45.705525 25081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "0730afde6e2846728e1f75b905798fd3"
format_stamp: "Formatted at 2026-06-19 02:48:00 on dist-test-slave-9tkz"
server_key: "312e372fd45dc97fd905cdd683f0926c"
server_key_iv: "36fdb27ef84d3e213759f8d714de1256"
server_key_version: "encryptionkey@0"
I20260619 02:48:45.706573 25081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:48:45.732216 25081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:48:45.735994 25081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:48:45.736256 25081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:48:45.737197 25081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:48:45.738476 25081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:48:45.738588 25081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.738708 25081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:48:45.738788 25081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:48:45.771760 25081 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:41571
I20260619 02:48:45.771845 25208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:41571 every 8 connection(s)
I20260619 02:48:45.773187 25081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:48:45.777803 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.776063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:60197)
0619 02:48:45.776734 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.776970 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
W20260619 02:48:45.778453 25209 heartbeater.cc:644] Failed to heartbeat to 127.7.246.126:34605 (0 consecutive failures): Network error: Failed to ping master at 127.7.246.126:34605: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
I20260619 02:48:45.779187 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.778910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:37103)
0619 02:48:45.779027 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.779091 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:48:45.779997 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.779731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:37799)
0619 02:48:45.779843 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.779906 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 02:48:45.780236 25209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:48:45.782001 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25081
I20260619 02:48:45.869458 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:45.868660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:54623)
0619 02:48:45.869101 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:45.869236 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":213,"threads_started":1}
I20260619 02:48:46.310329 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:46.309515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:46525)
0619 02:48:46.310014 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:46.310111 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":231,"threads_started":1}
I20260619 02:48:46.781426 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:46.780702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:54851)
0619 02:48:46.781136 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:46.781214 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":236,"threads_started":1}
I20260619 02:48:46.870934 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:46.870280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:42473)
0619 02:48:46.870687 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:46.870764 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":204,"threads_started":1}
I20260619 02:48:47.311883 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:47.311136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:36121)
0619 02:48:47.311615 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:47.311710 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":241,"threads_started":1}
I20260619 02:48:47.782848 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:47.782229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:53527)
0619 02:48:47.782625 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:47.782699 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":200,"threads_started":1}
I20260619 02:48:47.872555 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:47.871742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:49309)
0619 02:48:47.872240 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:47.872346 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260619 02:48:48.313623 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:48.312830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:45783)
0619 02:48:48.313311 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:48.313392 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260619 02:48:48.784355 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:48.783672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:36517)
0619 02:48:48.784120 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:48.784196 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260619 02:48:48.874132 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:48.873320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:53225)
0619 02:48:48.873819 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:48.873901 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260619 02:48:49.315068 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:49.314394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:46719)
0619 02:48:49.314815 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:49.314889 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":193,"threads_started":1}
I20260619 02:48:49.786079 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:49.785293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:49601)
0619 02:48:49.785794 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:49.785883 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":200,"threads_started":1}
I20260619 02:48:49.875844 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:49.875104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:47851)
0619 02:48:49.875543 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:49.875650 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":221,"threads_started":1}
I20260619 02:48:50.316682 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:50.316011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:60749)
0619 02:48:50.316430 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:50.316510 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":204,"threads_started":1}
I20260619 02:48:50.787850 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:50.787122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:37307)
0619 02:48:50.787608 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:50.787693 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260619 02:48:50.877818 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:50.876994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:54955)
0619 02:48:50.877532 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:50.877660 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260619 02:48:51.318501 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:51.317715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:52887)
0619 02:48:51.318220 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:51.318328 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
I20260619 02:48:51.789286 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:51.788634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:57473)
0619 02:48:51.789060 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:51.789140 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":244,"threads_started":1}
I20260619 02:48:51.879518 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:51.878781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:34171)
0619 02:48:51.879233 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:51.879309 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":204,"threads_started":1}
I20260619 02:48:52.320381 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:52.319401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:48419)
0619 02:48:52.319986 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:52.320087 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
I20260619 02:48:52.791024 25233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:52.790205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:44991)
0619 02:48:52.790703 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:52.790790 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":223,"threads_started":1}
I20260619 02:48:52.881321 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:52.880457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:43269)
0619 02:48:52.880997 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:52.881104 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":262,"threads_started":1}
I20260619 02:48:53.322158 25235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:53.321358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:53205)
0619 02:48:53.321917 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:53.322000 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":285,"threads_started":1}
I20260619 02:48:53.792524 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:53.791891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:57197)
0619 02:48:53.792292 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:53.792374 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":207,"threads_started":1}
I20260619 02:48:53.882917 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:53.882173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:43835)
0619 02:48:53.882615 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:53.882750 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":213,"threads_started":1}
I20260619 02:48:54.323700 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:54.322989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:45075)
0619 02:48:54.323452 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:54.323526 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260619 02:48:54.794154 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:54.793385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:46129)
0619 02:48:54.793917 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:54.793993 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":344,"threads_started":1}
I20260619 02:48:54.884670 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:54.883907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:40613)
0619 02:48:54.884383 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:54.884481 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260619 02:48:55.325399 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:55.324725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:54915)
0619 02:48:55.325152 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:55.325227 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260619 02:48:55.795893 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:55.795085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:52003)
0619 02:48:55.795646 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:55.795729 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":333,"threads_started":1}
I20260619 02:48:55.886420 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:55.885675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:54597)
0619 02:48:55.886152 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:55.886239 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260619 02:48:56.327071 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:56.326332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:36153)
0619 02:48:56.326775 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:56.326882 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":207,"threads_started":1}
I20260619 02:48:56.797530 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:56.796746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:49311)
0619 02:48:56.797199 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:56.797321 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260619 02:48:56.888078 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:56.887363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:35953)
0619 02:48:56.887846 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:56.887923 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":219,"threads_started":1}
I20260619 02:48:57.328848 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:57.328141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:53113)
0619 02:48:57.328564 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:57.328680 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260619 02:48:57.799039 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:57.798422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:60859)
0619 02:48:57.798809 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:57.798892 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":199,"threads_started":1}
I20260619 02:48:57.889732 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:57.888953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:34285)
0619 02:48:57.889387 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:57.889480 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260619 02:48:58.330651 25250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:58.329936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:52659)
0619 02:48:58.330383 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:58.330471 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":221,"threads_started":1}
I20260619 02:48:58.800721 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:58.799947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:39469)
0619 02:48:58.800420 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:58.800546 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":231,"threads_started":1}
I20260619 02:48:58.891438 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:58.890704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:42555)
0619 02:48:58.891152 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:58.891265 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":247,"threads_started":1}
I20260619 02:48:59.332127 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:59.331460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:55997)
0619 02:48:59.331879 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:59.331956 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260619 02:48:59.802196 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:59.801500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:50749)
0619 02:48:59.801940 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:59.802025 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260619 02:48:59.893129 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:48:59.892472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:52635)
0619 02:48:59.892877 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:48:59.892959 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":215,"threads_started":1}
I20260619 02:49:00.333998 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:00.333237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.66:43875)
0619 02:49:00.333711 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:00.333825 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":226,"threads_started":1}
I20260619 02:49:00.790350 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24779
I20260619 02:49:00.804078 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:00.803323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.67:60019)
0619 02:49:00.803802 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:00.803909 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":257,"threads_started":1}
I20260619 02:49:00.808531 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:00.791470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.7.246.65:44413)
0619 02:49:00.791988 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:00.792003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:00.792070 (+ 67us) server_negotiation.cc:407] Connection header received
0619 02:49:00.793755 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:00.793766 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:00.794197 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:00.794487 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:00.795985 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:00.797200 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:00.798476 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:00.799140 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:00.799268 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:00.801997 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:00.802016 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:00.802033 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:00.802082 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:00.805311 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:00.806029 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:00.806036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:00.806043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:00.806217 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:00.806556 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:00.806559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:00.806561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:00.807222 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:00.807445 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:00.807840 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 02:49:00.808259 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":223,"threads_started":1}
I20260619 02:49:00.810058 24942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:00.895802 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:00.894675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:34605 (local address 127.7.246.65:57449)
0619 02:49:00.895246 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:00.895493 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.126:34605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":271,"threads_started":1}
I20260619 02:49:00.897737 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24779
I20260619 02:49:00.915318 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 24950
I20260619 02:49:00.932298 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:00.916229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.7.246.66:46755)
0619 02:49:00.916832 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:00.916845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:00.916882 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:49:00.918621 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:00.918631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:00.919055 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:00.919348 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:00.920528 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:00.921854 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:00.923025 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:00.923647 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:00.923817 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:00.926333 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:00.926341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:00.926356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:00.926389 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:00.929299 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:00.930063 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:00.930070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:00.930075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:00.930250 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:00.930591 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:00.930594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:00.930596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:00.931114 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:00.931351 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:00.931661 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 02:49:00.932017 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":307,"threads_started":1}
I20260619 02:49:00.933681 25073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:01.016187 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 24950
I20260619 02:49:01.033382 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25081
I20260619 02:49:01.050084 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:01.034392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.7.246.67:41571)
0619 02:49:01.035055 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:01.035069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:01.035107 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:49:01.036699 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:01.036725 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:01.037154 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:01.037456 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:01.038767 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:01.039953 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:01.041147 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:01.041849 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:01.041972 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:01.044527 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:01.044534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:01.044547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:01.044576 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:01.047333 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:01.047918 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:01.047924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:01.047929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:01.048081 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:01.048395 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:01.048398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:01.048400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:01.048863 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:01.049076 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:01.049365 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:49:01.049801 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":368,"threads_started":1}
I20260619 02:49:01.051422 25204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:01.133347 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25081
2026-06-19T02:49:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64325 ms)
[----------] 30 tests from SecurityITest (293909 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:01 dist-test-slave-9tkz krb5kdc[25267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:01 dist-test-slave-9tkz krb5kdc[25267](info): set up 2 sockets
Jun 19 02:49:01 dist-test-slave-9tkz krb5kdc[25267](info): commencing operation
krb5kdc: starting...
W20260619 02:49:02.229720 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:02 dist-test-slave-9tkz krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:02.467314 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45655
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:02.797372 25280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:02.797676 25280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:02.797791 25280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:02.805054 25280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:02.805158 25280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:02.805202 25280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:02.805250 25280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:02.805292 25280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:02.815129 25280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:02.816398 25280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:02.817692 25280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:02.830272 25285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:02.830240 25286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:02.830242 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:02.830904 25280 server_base.cc:1034] running on GCE node
I20260619 02:49:02.831753 25280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:02.833156 25280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:02.834434 25280 hybrid_clock.cc:648] HybridClock initialized: now 1781837342834379 us; error 55 us; skew 500 ppm
I20260619 02:49:02.834854 25280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:02 dist-test-slave-9tkz krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:02.837936 25280 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:02.839499 25280 webserver.cc:460] Webserver started at http://127.7.246.126:38749/ using document root <none> and password file <none>
I20260619 02:49:02.840179 25280 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:02.840327 25280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:02.840633 25280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:02.843677 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff"
format_stamp: "Formatted at 2026-06-19 02:49:02 on dist-test-slave-9tkz"
server_key: "80c2c58a05b927674c65dc1ffeae2f5d"
server_key_iv: "e7a062c37999c0b78e32a63702daf7a5"
server_key_version: "encryptionkey@0"
I20260619 02:49:02.844440 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff"
format_stamp: "Formatted at 2026-06-19 02:49:02 on dist-test-slave-9tkz"
server_key: "80c2c58a05b927674c65dc1ffeae2f5d"
server_key_iv: "e7a062c37999c0b78e32a63702daf7a5"
server_key_version: "encryptionkey@0"
I20260619 02:49:02.849787 25280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:02.853214 25295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:02.854782 25280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:02.854974 25280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff"
format_stamp: "Formatted at 2026-06-19 02:49:02 on dist-test-slave-9tkz"
server_key: "80c2c58a05b927674c65dc1ffeae2f5d"
server_key_iv: "e7a062c37999c0b78e32a63702daf7a5"
server_key_version: "encryptionkey@0"
I20260619 02:49:02.855187 25280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:02.882283 25280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:02.886487 25280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:02.886807 25280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:02.905453 25280 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:45655
I20260619 02:49:02.905471 25346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:45655 every 8 connection(s)
I20260619 02:49:02.906842 25280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:02.910374 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25280
I20260619 02:49:02.910622 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:02.910687 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:02.911275 8153 external_mini_cluster.cc:1428] Setting key aae8efa02f930d4d664ff635d4840577
I20260619 02:49:02.924496 25347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: Bootstrap starting.
Jun 19 02:49:02 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:02.927947 25347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:02.928957 25347 log.cc:824] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:02.932124 25347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: No bootstrap required, opened a new log
I20260619 02:49:02.934562 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:02.913195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.7.246.126:45655)
0619 02:49:02.914023 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:02.914037 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:02.914085 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:02.915675 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:02.915708 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:02.916201 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:02.916626 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:02.918046 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:02.919969 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:02.921336 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:02.922443 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:02.922629 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:02.926913 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:02.926921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:02.926934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:02.926969 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:02.930485 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:02.931319 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:02.931325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:02.931331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:02.931491 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:02.931926 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:02.931929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:02.931931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:02.932432 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:02.932660 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:02.933139 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 02:49:02.933509 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":342,"threads_started":1}
I20260619 02:49:02.940187 25347 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } }
I20260619 02:49:02.940572 25347 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:02.940670 25347 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f82b2f95f7461d94c54d22e2d0b0ff, State: Initialized, Role: FOLLOWER
I20260619 02:49:02.941200 25347 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } }
I20260619 02:49:02.941429 25347 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:02.941577 25347 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:02.941758 25347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:02.943943 25347 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } }
I20260619 02:49:02.944440 25347 leader_election.cc:302] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8f82b2f95f7461d94c54d22e2d0b0ff; no voters:
I20260619 02:49:02.945076 25347 leader_election.cc:288] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:02.945253 25352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:02.946412 25352 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [term 1 LEADER]: Becoming Leader. State: Replica: f8f82b2f95f7461d94c54d22e2d0b0ff, State: Running, Role: LEADER
I20260619 02:49:02.947046 25352 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } }
I20260619 02:49:02.947901 25347 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:02.950112 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8f82b2f95f7461d94c54d22e2d0b0ff. Latest consensus state: current_term: 1 leader_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } } }
I20260619 02:49:02.950199 25353 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8f82b2f95f7461d94c54d22e2d0b0ff" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45655 } } }
I20260619 02:49:02.950513 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:02.950599 25353 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:02.952020 25357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:02.957300 25357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:02.966399 25357 catalog_manager.cc:1347] Generated new cluster ID: 92207b1fc9fd48bf9abd901aaeddedec
I20260619 02:49:02.966535 25357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:02.978737 25357 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:02.980015 25357 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:02.985111 25357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: Generated new TSK 0
I20260619 02:49:02.985924 25357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:49:03.043006 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:03.382534 25373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:03.382799 25373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:03.382889 25373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:03.389861 25373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:03.390025 25373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:03.390195 25373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:03.399626 25373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:03.400890 25373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:03.402169 25373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:03.414937 25378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:03.414958 25379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:03.414872 25381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:03.415714 25373 server_base.cc:1034] running on GCE node
I20260619 02:49:03.416334 25373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:03.417217 25373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:03.418440 25373 hybrid_clock.cc:648] HybridClock initialized: now 1781837343418385 us; error 51 us; skew 500 ppm
I20260619 02:49:03.418738 25373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:03 dist-test-slave-9tkz krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:03.421743 25373 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:49:03.423079 25373 webserver.cc:460] Webserver started at http://127.7.246.65:33059/ using document root <none> and password file <none>
I20260619 02:49:03.423733 25373 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:03.423877 25373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:03.424185 25373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:03.427237 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "54159614ed7d45c18345a53eefbf5c80"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "921d6c490bf9c5f56e591646873d41c8"
server_key_iv: "e2005e6ea4d7f25c698a373b04e977f3"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.427994 25373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "54159614ed7d45c18345a53eefbf5c80"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "921d6c490bf9c5f56e591646873d41c8"
server_key_iv: "e2005e6ea4d7f25c698a373b04e977f3"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.432865 25373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:49:03.436272 25388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:03.437515 25373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:49:03.437704 25373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "54159614ed7d45c18345a53eefbf5c80"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "921d6c490bf9c5f56e591646873d41c8"
server_key_iv: "e2005e6ea4d7f25c698a373b04e977f3"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.437896 25373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:03.454138 25373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:03.458138 25373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:03.458484 25373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:03.459468 25373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:03.460738 25373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:03.460807 25373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:03.460898 25373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:03.460943 25373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:03.494215 25373 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:34697
I20260619 02:49:03.494309 25500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:34697 every 8 connection(s)
I20260619 02:49:03.495568 25373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:03.498957 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25373
I20260619 02:49:03.499195 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:03.500065 8153 external_mini_cluster.cc:1428] Setting key b837466321d3efdf44733c6cad176be2
Jun 19 02:49:03 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:03.519716 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:03.498277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:40189 (local address 127.7.246.126:45655)
0619 02:49:03.498752 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:03.498759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:03.499986 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 02:49:03.502033 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:03.502038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:03.502118 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:03.502206 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:03.504347 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:03.505050 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:03.507529 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:03.507949 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:03.508062 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:03.512427 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:03.512448 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:03.512456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:03.512500 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:03.516179 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:03.516954 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:03.516962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:03.516968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:03.517115 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:03.517626 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:03.517633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:03.517640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:03.518034 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:03.518191 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:03.518934 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 02:49:03.519204 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260619 02:49:03.520331 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:03.498776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45655 (local address 127.7.246.65:40189)
0619 02:49:03.499777 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:03.499839 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:03.501794 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:03.502494 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:03.502506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:03.503313 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:03.504174 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:03.504203 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:03.505226 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:03.505235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:03.507356 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:03.507373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:03.508207 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:03.508216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:03.508560 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:03.509296 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:03.509329 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:03.512194 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:03.516339 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:03.516345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:03.516359 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:03.516813 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:03.517250 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:03.517254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:03.517256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:03.517446 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:03.518177 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:03.518183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:03.518524 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 02:49:03.518765 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 02:49:03.519191 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":384,"threads_started":1}
I20260619 02:49:03.521978 25501 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45655
I20260619 02:49:03.522284 25501 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:03.523007 25501 heartbeater.cc:511] Master 127.7.246.126:45655 requested a full tablet report, sending...
I20260619 02:49:03.525575 25312 ts_manager.cc:194] Registered new tserver with Master: 54159614ed7d45c18345a53eefbf5c80 (127.7.246.65:34697)
I20260619 02:49:03.527731 25312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:40189
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:49:03.558696 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:03.881011 25507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:03.881276 25507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:03.881439 25507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:03.888669 25507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:03.888777 25507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:03.888949 25507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:03.898752 25507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:03.900017 25507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:03.901336 25507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:03.915210 25513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:03.915424 25512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:03.915654 25515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:03.916000 25507 server_base.cc:1034] running on GCE node
I20260619 02:49:03.916591 25507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:03.917688 25507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:03.918953 25507 hybrid_clock.cc:648] HybridClock initialized: now 1781837343918910 us; error 34 us; skew 500 ppm
I20260619 02:49:03.919301 25507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:03 dist-test-slave-9tkz krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:03.922731 25507 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:49:03.924898 25507 webserver.cc:460] Webserver started at http://127.7.246.66:41241/ using document root <none> and password file <none>
I20260619 02:49:03.925678 25507 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:03.925836 25507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:03.926157 25507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:03.929108 25507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "fe7e74d5cb874986a536f8157fad4bf4"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "d5e39e9410e0c37d8524904d09e46804"
server_key_iv: "e9920be8ec9c601ccdec7115c6724d7b"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.929934 25507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "fe7e74d5cb874986a536f8157fad4bf4"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "d5e39e9410e0c37d8524904d09e46804"
server_key_iv: "e9920be8ec9c601ccdec7115c6724d7b"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.934934 25507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:49:03.938550 25522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:03.939790 25507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 02:49:03.939960 25507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "fe7e74d5cb874986a536f8157fad4bf4"
format_stamp: "Formatted at 2026-06-19 02:49:03 on dist-test-slave-9tkz"
server_key: "d5e39e9410e0c37d8524904d09e46804"
server_key_iv: "e9920be8ec9c601ccdec7115c6724d7b"
server_key_version: "encryptionkey@0"
I20260619 02:49:03.940147 25507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:03.963261 25507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:03.967271 25507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:03.967564 25507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:03.968531 25507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:03.969923 25507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:03.970024 25507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:03.970170 25507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:03.970243 25507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:04.004068 25507 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:39359
I20260619 02:49:04.004113 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:39359 every 8 connection(s)
I20260619 02:49:04.005376 25507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:04.011675 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25507
I20260619 02:49:04.011883 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:04.012315 8153 external_mini_cluster.cc:1428] Setting key ffc9b4be3acae957af0eba6723ce422e
Jun 19 02:49:04 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837343, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:04.025446 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.007929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:57217 (local address 127.7.246.126:45655)
0619 02:49:04.008096 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.008100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.009157 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 02:49:04.011025 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.011033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.011127 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.011237 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:04.012852 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.013769 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.015310 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.015726 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.015869 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.019268 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:04.019273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:04.019276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:04.019306 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:04.022491 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.023192 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.023198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.023202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.023340 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.023774 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.023784 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.023786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.024057 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:04.024207 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.024877 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.025098 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:49:04.026592 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.008352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45655 (local address 127.7.246.66:57217)
0619 02:49:04.008997 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.009041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.010763 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.011463 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.011471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.012064 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:04.012688 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.012705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.013926 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.013954 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.015169 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.015180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.015936 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.015942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.016184 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.016870 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:04.016893 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:04.019054 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:04.022623 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:04.022632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:04.022649 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:04.023072 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:04.023464 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:04.023468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:04.023472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:04.023653 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:04.024197 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:04.024204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:04.024534 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.024792 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.025320 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
I20260619 02:49:04.028146 25635 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45655
I20260619 02:49:04.028443 25635 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:04.029165 25635 heartbeater.cc:511] Master 127.7.246.126:45655 requested a full tablet report, sending...
I20260619 02:49:04.030813 25312 ts_manager.cc:194] Registered new tserver with Master: fe7e74d5cb874986a536f8157fad4bf4 (127.7.246.66:39359)
I20260619 02:49:04.032086 25312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:57217
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:49:04.066280 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:36179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:04.391528 25640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:04.391912 25640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:04.392103 25640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:04.399294 25640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:04.399386 25640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:04.399581 25640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:04.409042 25640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:36179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:45655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:04.410298 25640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:04.411717 25640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:04.424587 25646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:04.424677 25648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:04.424741 25645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:04.426417 25640 server_base.cc:1034] running on GCE node
I20260619 02:49:04.427021 25640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:04.427881 25640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:04.429092 25640 hybrid_clock.cc:648] HybridClock initialized: now 1781837344429038 us; error 50 us; skew 500 ppm
I20260619 02:49:04.429404 25640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:04 dist-test-slave-9tkz krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837344, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:04.432456 25640 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:49:04.433907 25640 webserver.cc:460] Webserver started at http://127.7.246.67:36389/ using document root <none> and password file <none>
I20260619 02:49:04.434593 25640 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:04.434744 25640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:04.435041 25640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:04.438380 25640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "e4011dc22c7a403cab9cfb008491348f"
format_stamp: "Formatted at 2026-06-19 02:49:04 on dist-test-slave-9tkz"
server_key: "0e40f5bc8627f3265ba43de23e92980b"
server_key_iv: "a11a9cd7eead5a4f509f5ad5b82e6834"
server_key_version: "encryptionkey@0"
I20260619 02:49:04.439332 25640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "e4011dc22c7a403cab9cfb008491348f"
format_stamp: "Formatted at 2026-06-19 02:49:04 on dist-test-slave-9tkz"
server_key: "0e40f5bc8627f3265ba43de23e92980b"
server_key_iv: "a11a9cd7eead5a4f509f5ad5b82e6834"
server_key_version: "encryptionkey@0"
I20260619 02:49:04.445906 25640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 02:49:04.449769 25655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:04.451148 25640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:04.451361 25640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "e4011dc22c7a403cab9cfb008491348f"
format_stamp: "Formatted at 2026-06-19 02:49:04 on dist-test-slave-9tkz"
server_key: "0e40f5bc8627f3265ba43de23e92980b"
server_key_iv: "a11a9cd7eead5a4f509f5ad5b82e6834"
server_key_version: "encryptionkey@0"
I20260619 02:49:04.451597 25640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:04.468398 25640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:04.472260 25640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:04.472549 25640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:04.473634 25640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:04.474938 25640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:04.475045 25640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:04.475199 25640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:04.475272 25640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:04.508684 25640 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:33543
I20260619 02:49:04.508737 25767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:33543 every 8 connection(s)
I20260619 02:49:04.510223 25640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:04.520455 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25640
I20260619 02:49:04.520664 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:04.521204 8153 external_mini_cluster.cc:1428] Setting key 246adf96ac0dd90c718e17c814b8b221
Jun 19 02:49:04 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837344, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:04.530058 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.512655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:35661 (local address 127.7.246.126:45655)
0619 02:49:04.512836 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.512841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.513953 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 02:49:04.515745 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.515750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.515817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.515905 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:04.517315 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.518120 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.519670 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.520120 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.520263 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.524089 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:04.524094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:04.524098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:04.524164 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:04.526831 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.527721 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.527725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.527728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.527860 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.528284 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.528290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.528294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.528576 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:04.528729 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.529429 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.529678 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:49:04.530942 25501 heartbeater.cc:503] Master 127.7.246.126:45655 was elected leader, sending a full tablet report...
I20260619 02:49:04.531054 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.513088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:45655 (local address 127.7.246.67:35661)
0619 02:49:04.513792 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.513839 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.515470 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.516108 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.516116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.516623 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:04.517173 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.517191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.518259 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.518263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.519536 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.519549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.520377 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.520382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.520595 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.521440 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:04.521471 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:04.523872 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:04.526990 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:04.527000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:04.527018 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:04.527579 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:04.527971 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:04.527976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:04.527980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:04.528173 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:04.528701 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:04.528710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:04.529069 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.529345 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.529925 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":247,"threads_started":1}
I20260619 02:49:04.532701 25768 heartbeater.cc:347] Connected to a master server at 127.7.246.126:45655
I20260619 02:49:04.533020 25768 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:04.533838 25768 heartbeater.cc:511] Master 127.7.246.126:45655 requested a full tablet report, sending...
I20260619 02:49:04.535488 25312 ts_manager.cc:194] Registered new tserver with Master: e4011dc22c7a403cab9cfb008491348f (127.7.246.67:33543)
I20260619 02:49:04.536576 25312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:35661
I20260619 02:49:04.536823 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:04.554013 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.541495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.7.246.126:45655)
0619 02:49:04.541708 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.541717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.542184 (+ 467us) server_negotiation.cc:407] Connection header received
0619 02:49:04.542321 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.542324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.542387 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.542471 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:04.543998 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.544711 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.546146 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.546536 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.546630 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.549012 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:04.549016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:04.549019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:04.549048 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:04.551493 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.552105 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.552109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.552112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.552234 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.552563 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.552566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.552568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.552878 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:04.553055 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.553341 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.553625 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:49:04.558033 25312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:04.564987 25312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:04.605921 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.576798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:34697 (local address 127.0.0.1:47998)
0619 02:49:04.596476 (+ 19678us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.596534 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.596765 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.597833 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.597840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.597888 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:04.598355 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.598367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.600465 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.600470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.603448 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.603490 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.604811 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.604816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.604992 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.605045 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.605213 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.605404 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19329,"mutex_wait_us":86,"thread_start_us":19166,"threads_started":1}
I20260619 02:49:04.606168 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.578004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.7.246.67:33543)
0619 02:49:04.595719 (+ 17715us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.595732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.597848 (+ 2116us) server_negotiation.cc:407] Connection header received
0619 02:49:04.598172 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.598181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.598462 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.598668 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:04.599334 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.600932 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.603635 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.605062 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.605188 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.605406 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.605624 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.605796 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17515,"thread_start_us":17368,"threads_started":1}
I20260619 02:49:04.606769 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.577126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.7.246.65:34697)
0619 02:49:04.594917 (+ 17791us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.594927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.597035 (+ 2108us) server_negotiation.cc:407] Connection header received
0619 02:49:04.597174 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.597186 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.597539 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.598410 (+ 871us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:04.598668 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.600288 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.603678 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.604619 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.604779 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.605034 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.606409 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.606537 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17586,"thread_start_us":17809,"threads_started":1}
I20260619 02:49:04.606106 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.577365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33543 (local address 127.0.0.1:45954)
0619 02:49:04.597687 (+ 20322us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.597722 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.597962 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.598650 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.598655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.598679 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:04.599114 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.599133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.601105 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.601110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.603487 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.603497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.605268 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.605274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.605429 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.605458 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.605575 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.605693 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20142,"mutex_wait_us":92,"thread_start_us":20359,"threads_started":1}
I20260619 02:49:04.606606 25782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.577635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39359 (local address 127.0.0.1:33786)
0619 02:49:04.598052 (+ 20417us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.598069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.598187 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.598952 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.598961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.598989 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:04.599389 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.599397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.601439 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.601445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.603783 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.603796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.605807 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.605815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.605992 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.606021 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.606132 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.606262 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20260,"mutex_wait_us":72,"thread_start_us":19733,"threads_started":1}
I20260619 02:49:04.607106 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.578287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.7.246.66:39359)
0619 02:49:04.597175 (+ 18888us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.597188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.598197 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 02:49:04.598350 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.598358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.598601 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.598813 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:04.599524 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.601301 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.603957 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.605290 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.605430 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.605646 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.606697 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.606816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18739,"thread_start_us":18641,"threads_started":1}
I20260619 02:49:04.610598 25570 tablet_service.cc:1467] Processing CreateTablet for tablet 30e899dbad1247ec8f038a3b27803f03 (DEFAULT_TABLE table=test-table [id=f4bcb918a0f84042b43769c043b10b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:04.610580 25436 tablet_service.cc:1467] Processing CreateTablet for tablet 30e899dbad1247ec8f038a3b27803f03 (DEFAULT_TABLE table=test-table [id=f4bcb918a0f84042b43769c043b10b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:04.611886 25570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e899dbad1247ec8f038a3b27803f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:04.612004 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e899dbad1247ec8f038a3b27803f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:04.612648 25703 tablet_service.cc:1467] Processing CreateTablet for tablet 30e899dbad1247ec8f038a3b27803f03 (DEFAULT_TABLE table=test-table [id=f4bcb918a0f84042b43769c043b10b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:04.613951 25703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e899dbad1247ec8f038a3b27803f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:04.623189 25786 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Bootstrap starting.
I20260619 02:49:04.626386 25786 tablet_bootstrap.cc:654] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:04.627374 25787 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Bootstrap starting.
I20260619 02:49:04.627661 25786 log.cc:824] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:04.630125 25788 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Bootstrap starting.
I20260619 02:49:04.631012 25787 tablet_bootstrap.cc:654] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:04.632335 25787 log.cc:824] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:04.632473 25786 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: No bootstrap required, opened a new log
I20260619 02:49:04.632714 25786 ts_tablet_manager.cc:1397] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 02:49:04.634702 25788 tablet_bootstrap.cc:654] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:04.635502 25787 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: No bootstrap required, opened a new log
I20260619 02:49:04.635811 25787 ts_tablet_manager.cc:1397] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:49:04.635969 25788 log.cc:824] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:04.638942 25788 tablet_bootstrap.cc:492] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: No bootstrap required, opened a new log
I20260619 02:49:04.639183 25788 ts_tablet_manager.cc:1397] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:49:04.640908 25786 raft_consensus.cc:348] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.641294 25786 raft_consensus.cc:374] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:04.641388 25786 raft_consensus.cc:729] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7e74d5cb874986a536f8157fad4bf4, State: Initialized, Role: FOLLOWER
I20260619 02:49:04.642194 25786 consensus_queue.cc:260] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.646282 25787 raft_consensus.cc:348] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.646685 25787 raft_consensus.cc:374] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:04.646797 25787 raft_consensus.cc:729] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54159614ed7d45c18345a53eefbf5c80, State: Initialized, Role: FOLLOWER
I20260619 02:49:04.647344 25787 consensus_queue.cc:260] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.649152 25787 ts_tablet_manager.cc:1428] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 02:49:04.649840 25788 raft_consensus.cc:348] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.650131 25635 heartbeater.cc:503] Master 127.7.246.126:45655 was elected leader, sending a full tablet report...
I20260619 02:49:04.650279 25788 raft_consensus.cc:374] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:04.650422 25788 raft_consensus.cc:729] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4011dc22c7a403cab9cfb008491348f, State: Initialized, Role: FOLLOWER
I20260619 02:49:04.650792 25786 ts_tablet_manager.cc:1428] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260619 02:49:04.651091 25788 consensus_queue.cc:260] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.652196 25768 heartbeater.cc:503] Master 127.7.246.126:45655 was elected leader, sending a full tablet report...
I20260619 02:49:04.652793 25788 ts_tablet_manager.cc:1428] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260619 02:49:04.726933 25793 raft_consensus.cc:482] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:04.727285 25793 raft_consensus.cc:504] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.729218 25793 leader_election.cc:288] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4011dc22c7a403cab9cfb008491348f (127.7.246.67:33543), fe7e74d5cb874986a536f8157fad4bf4 (127.7.246.66:39359)
I20260619 02:49:04.737330 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.729902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52411 (local address 127.7.246.67:33543)
0619 02:49:04.730125 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.730134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.730826 (+ 692us) server_negotiation.cc:407] Connection header received
0619 02:49:04.731045 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.731051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.731130 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.731231 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:04.732104 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.733095 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.735429 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.736206 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.736362 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.736459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.736949 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.737069 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:49:04.737403 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.729857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:32873 (local address 127.7.246.66:39359)
0619 02:49:04.730105 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.730114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.730820 (+ 706us) server_negotiation.cc:407] Connection header received
0619 02:49:04.731047 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.731052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.731130 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.731231 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:04.732228 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.733297 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.735245 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.736311 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.736504 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.736602 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.736927 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.737073 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:49:04.737506 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.729452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:39359 (local address 127.7.246.65:32873)
0619 02:49:04.730633 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.730673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.730907 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.731290 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.731296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.731345 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:04.732048 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.732063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.733449 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.733453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.735069 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.735081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.736566 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.736574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.736741 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.736786 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.736889 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.737017 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":75,"spinlock_wait_cycles":60544,"thread_start_us":862,"threads_started":1}
I20260619 02:49:04.737623 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.729452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:33543 (local address 127.7.246.65:52411)
0619 02:49:04.730706 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:04.730726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:04.730870 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:04.731290 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:04.731296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:04.731332 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:04.731892 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.731908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.733248 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.733256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.735267 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:04.735284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.736468 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.736475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:04.736737 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.736780 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:49:04.736910 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:49:04.737062 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":121,"thread_start_us":549,"threads_started":1}
I20260619 02:49:04.738693 25590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e899dbad1247ec8f038a3b27803f03" candidate_uuid: "54159614ed7d45c18345a53eefbf5c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7e74d5cb874986a536f8157fad4bf4" is_pre_election: true
I20260619 02:49:04.738850 25723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e899dbad1247ec8f038a3b27803f03" candidate_uuid: "54159614ed7d45c18345a53eefbf5c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4011dc22c7a403cab9cfb008491348f" is_pre_election: true
I20260619 02:49:04.739225 25590 raft_consensus.cc:2454] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54159614ed7d45c18345a53eefbf5c80 in term 0.
I20260619 02:49:04.739501 25723 raft_consensus.cc:2454] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54159614ed7d45c18345a53eefbf5c80 in term 0.
I20260619 02:49:04.739928 25390 leader_election.cc:302] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54159614ed7d45c18345a53eefbf5c80, fe7e74d5cb874986a536f8157fad4bf4; no voters:
I20260619 02:49:04.740340 25793 raft_consensus.cc:2781] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:04.740490 25793 raft_consensus.cc:482] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:04.740609 25793 raft_consensus.cc:3037] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:04.742911 25793 raft_consensus.cc:504] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.743551 25793 leader_election.cc:288] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [CANDIDATE]: Term 1 election: Requested vote from peers e4011dc22c7a403cab9cfb008491348f (127.7.246.67:33543), fe7e74d5cb874986a536f8157fad4bf4 (127.7.246.66:39359)
I20260619 02:49:04.743997 25590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e899dbad1247ec8f038a3b27803f03" candidate_uuid: "54159614ed7d45c18345a53eefbf5c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7e74d5cb874986a536f8157fad4bf4"
I20260619 02:49:04.744061 25723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e899dbad1247ec8f038a3b27803f03" candidate_uuid: "54159614ed7d45c18345a53eefbf5c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4011dc22c7a403cab9cfb008491348f"
I20260619 02:49:04.744182 25590 raft_consensus.cc:3037] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:04.744338 25723 raft_consensus.cc:3037] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:04.746528 25590 raft_consensus.cc:2454] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54159614ed7d45c18345a53eefbf5c80 in term 1.
I20260619 02:49:04.746922 25390 leader_election.cc:302] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54159614ed7d45c18345a53eefbf5c80, fe7e74d5cb874986a536f8157fad4bf4; no voters:
I20260619 02:49:04.747211 25793 raft_consensus.cc:2781] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:04.747215 25723 raft_consensus.cc:2454] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54159614ed7d45c18345a53eefbf5c80 in term 1.
W20260619 02:49:04.747872 25502 tablet.cc:2369] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:04.748605 25793 raft_consensus.cc:686] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 1 LEADER]: Becoming Leader. State: Replica: 54159614ed7d45c18345a53eefbf5c80, State: Running, Role: LEADER
I20260619 02:49:04.749053 25793 consensus_queue.cc:237] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } }
I20260619 02:49:04.753891 25310 catalog_manager.cc:5495] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54159614ed7d45c18345a53eefbf5c80 (127.7.246.65). New cstate: current_term: 1 leader_uuid: "54159614ed7d45c18345a53eefbf5c80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54159614ed7d45c18345a53eefbf5c80" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 } health_report { overall_health: UNKNOWN } } }
W20260619 02:49:04.757390 25636 tablet.cc:2369] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:04.761781 25769 tablet.cc:2369] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:04.781697 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.776476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.7.246.65:34697)
0619 02:49:04.776682 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.776687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.776705 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:49:04.776901 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.776905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.776981 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.777125 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:04.777484 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.778214 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.779601 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.780004 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.780135 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.780624 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:04.781096 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:04.781246 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.781338 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.781415 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:49:04.789862 25590 raft_consensus.cc:1261] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 1 FOLLOWER]: Refusing update from remote peer 54159614ed7d45c18345a53eefbf5c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:04.789861 25723 raft_consensus.cc:1261] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 1 FOLLOWER]: Refusing update from remote peer 54159614ed7d45c18345a53eefbf5c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:04.790813 25797 consensus_queue.cc:1035] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7e74d5cb874986a536f8157fad4bf4" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:04.791189 25793 consensus_queue.cc:1035] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4011dc22c7a403cab9cfb008491348f" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:04.803709 25801 mvcc.cc:205] Tried to move back new op lower bound from 7298405764249858048 to 7298405764096192512. Current Snapshot: MvccSnapshot[applied={T|T < 7298405764249858048}]
I20260619 02:49:04.814894 25312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:04.815212 25312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:04.818634 25312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8f82b2f95f7461d94c54d22e2d0b0ff: Sending DeleteTablet for 3 replicas of tablet 30e899dbad1247ec8f038a3b27803f03
I20260619 02:49:04.819720 25436 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e899dbad1247ec8f038a3b27803f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:04 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47998
I20260619 02:49:04.819748 25703 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e899dbad1247ec8f038a3b27803f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:04 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:45954
I20260619 02:49:04.820465 25570 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e899dbad1247ec8f038a3b27803f03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:04 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:33786
I20260619 02:49:04.821444 25812 tablet_replica.cc:330] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: stopping tablet replica
I20260619 02:49:04.821761 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25373
I20260619 02:49:04.821995 25812 raft_consensus.cc:2229] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:04.822546 25812 raft_consensus.cc:2258] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:04.826607 25812 ts_tablet_manager.cc:1905] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:04.829139 25811 tablet_replica.cc:330] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: stopping tablet replica
I20260619 02:49:04.829728 25811 raft_consensus.cc:2229] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:04.830394 25811 raft_consensus.cc:2258] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:04.833006 25810 tablet_replica.cc:330] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: stopping tablet replica
I20260619 02:49:04.833441 25810 raft_consensus.cc:2229] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:04.833714 25812 ts_tablet_manager.cc:1918] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:04.833873 25812 log.cc:1196] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/30e899dbad1247ec8f038a3b27803f03
I20260619 02:49:04.834136 25810 raft_consensus.cc:2258] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:04.834414 25812 ts_tablet_manager.cc:1939] T 30e899dbad1247ec8f038a3b27803f03 P fe7e74d5cb874986a536f8157fad4bf4: Deleting consensus metadata
I20260619 02:49:04.835633 25811 ts_tablet_manager.cc:1905] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:04.836156 25810 ts_tablet_manager.cc:1905] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:04.836249 25297 catalog_manager.cc:4841] TS fe7e74d5cb874986a536f8157fad4bf4 (127.7.246.66:39359): tablet 30e899dbad1247ec8f038a3b27803f03 (table test-table [id=f4bcb918a0f84042b43769c043b10b8c]) successfully deleted
Jun 19 02:49:04 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:49:04.844203 25810 ts_tablet_manager.cc:1918] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:04.844380 25810 log.cc:1196] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/30e899dbad1247ec8f038a3b27803f03
I20260619 02:49:04.844957 25810 ts_tablet_manager.cc:1939] T 30e899dbad1247ec8f038a3b27803f03 P 54159614ed7d45c18345a53eefbf5c80: Deleting consensus metadata
I20260619 02:49:04.844936 25811 ts_tablet_manager.cc:1918] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:04.845223 25811 log.cc:1196] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/30e899dbad1247ec8f038a3b27803f03
I20260619 02:49:04.845809 25811 ts_tablet_manager.cc:1939] T 30e899dbad1247ec8f038a3b27803f03 P e4011dc22c7a403cab9cfb008491348f: Deleting consensus metadata
I20260619 02:49:04.847008 25299 catalog_manager.cc:4841] TS 54159614ed7d45c18345a53eefbf5c80 (127.7.246.65:34697): tablet 30e899dbad1247ec8f038a3b27803f03 (table test-table [id=f4bcb918a0f84042b43769c043b10b8c]) successfully deleted
I20260619 02:49:04.847584 25296 catalog_manager.cc:4841] TS e4011dc22c7a403cab9cfb008491348f (127.7.246.67:33543): tablet 30e899dbad1247ec8f038a3b27803f03 (table test-table [id=f4bcb918a0f84042b43769c043b10b8c]) successfully deleted
I20260619 02:49:04.851859 25781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.824552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.7.246.65:34697)
0619 02:49:04.825860 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.825869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.825894 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:49:04.825967 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.825975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.826078 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.826197 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:04.830945 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.832594 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.838000 (+ 5406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.838803 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.838911 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.844859 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:04.844871 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:04.844891 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:04.844951 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:04.848872 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.849713 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.849720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.849723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.849868 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.850279 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.850283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.850285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.850653 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:04.850978 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.851263 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.851502 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962}
I20260619 02:49:04.852591 25496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:04.950856 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25373
I20260619 02:49:04.970206 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25507
Jun 19 02:49:04 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:49:04.984965 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:04.970967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.7.246.66:39359)
0619 02:49:04.971131 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:04.971136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:04.971156 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:49:04.971304 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:04.971308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:04.971372 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:04.971457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:04.972717 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.973438 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.974769 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:04.975122 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:04.975209 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:04.979320 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:04.979338 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:04.979354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:04.979393 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:04.982109 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.982852 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.982859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.982861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.982991 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:04.983378 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:04.983383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:04.983389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:04.983774 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:04.984082 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:04.984376 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:49:04.984602 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:49:04.985698 25630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:05.075955 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25507
I20260619 02:49:05.092968 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25640
Jun 19 02:49:05 dist-test-slave-9tkz krb5kdc[25267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:49:05.108157 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:05.093842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.7.246.67:33543)
0619 02:49:05.094040 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:05.094046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:05.094068 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:49:05.094175 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:05.094181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:05.094283 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:05.094396 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:05.095705 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:05.096455 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:05.097972 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:05.098384 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:05.098530 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:05.102813 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:05.102828 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:05.102843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:05.102877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:05.105365 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:05.106129 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:05.106145 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:05.106147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:05.106288 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:05.106660 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:05.106665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:05.106670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:05.107014 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:05.107323 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:05.107610 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:49:05.107853 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:49:05.108829 25763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:05.197273 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25640
I20260619 02:49:05.213706 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25280
I20260619 02:49:05.214812 25342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:05.325258 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25280
2026-06-19T02:49:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T02:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:05Z Disabled control of system clock
I20260619 02:49:05.398859 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36719
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:05.727423 25821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:05.727748 25821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:05.727878 25821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:05.734936 25821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:05.735040 25821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:05.735077 25821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:05.735126 25821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:05.735170 25821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:05.744598 25821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:05.745959 25821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:05.747723 25821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:05.759119 25826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:05.759861 25827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:05.759884 25829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:05.760282 25821 server_base.cc:1034] running on GCE node
I20260619 02:49:05.761062 25821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:05.762578 25821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:05.763827 25821 hybrid_clock.cc:648] HybridClock initialized: now 1781837345763795 us; error 96 us; skew 500 ppm
I20260619 02:49:05.764185 25821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:05.770232 25821 webserver.cc:460] Webserver started at http://127.7.246.126:33271/ using document root <none> and password file <none>
I20260619 02:49:05.770952 25821 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:05.771085 25821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:05.771410 25821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:05.774417 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "8bc420e871dc491eb7845c26499f96e2"
format_stamp: "Formatted at 2026-06-19 02:49:05 on dist-test-slave-9tkz"
server_key: "3af24ab0feae508a148728a1d206c873"
server_key_iv: "54c70627c7dd0ff288215f2123247154"
server_key_version: "encryptionkey@0"
I20260619 02:49:05.775206 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "8bc420e871dc491eb7845c26499f96e2"
format_stamp: "Formatted at 2026-06-19 02:49:05 on dist-test-slave-9tkz"
server_key: "3af24ab0feae508a148728a1d206c873"
server_key_iv: "54c70627c7dd0ff288215f2123247154"
server_key_version: "encryptionkey@0"
I20260619 02:49:05.780270 25821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:05.783612 25835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:05.784934 25821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:05.785109 25821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "8bc420e871dc491eb7845c26499f96e2"
format_stamp: "Formatted at 2026-06-19 02:49:05 on dist-test-slave-9tkz"
server_key: "3af24ab0feae508a148728a1d206c873"
server_key_iv: "54c70627c7dd0ff288215f2123247154"
server_key_version: "encryptionkey@0"
I20260619 02:49:05.785310 25821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:05.809347 25821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:05.810312 25821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:05.810611 25821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:05.829247 25821 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:36719
I20260619 02:49:05.829270 25886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:36719 every 8 connection(s)
I20260619 02:49:05.830646 25821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:05.834477 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:05.839964 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25821
I20260619 02:49:05.840142 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:05.840592 8153 external_mini_cluster.cc:1428] Setting key 10d8609ad4847aa03ead028bf82ce259
I20260619 02:49:05.846426 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: Bootstrap starting.
I20260619 02:49:05.850386 25887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:05.851468 25887 log.cc:824] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:05.854265 25887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: No bootstrap required, opened a new log
I20260619 02:49:05.856134 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:05.842624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.7.246.126:36719)
0619 02:49:05.843806 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:05.843821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:05.843871 (+ 50us) server_negotiation.cc:407] Connection header received
0619 02:49:05.846276 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:05.846305 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:05.846895 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:05.847336 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:05.847879 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:05.849975 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:05.851390 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:05.852413 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:05.852625 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:05.853842 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:05.853850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:05.853863 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:05.854422 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:05.854521 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:05.854543 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:05.854728 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:05.854906 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 02:49:05.855206 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":400,"threads_started":1}
I20260619 02:49:05.863083 25887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } }
I20260619 02:49:05.863451 25887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:05.863565 25887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc420e871dc491eb7845c26499f96e2, State: Initialized, Role: FOLLOWER
I20260619 02:49:05.864120 25887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } }
I20260619 02:49:05.864352 25887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:05.864498 25887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:05.864678 25887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:05.866950 25887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } }
I20260619 02:49:05.867424 25887 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc420e871dc491eb7845c26499f96e2; no voters:
I20260619 02:49:05.868111 25887 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:05.868276 25892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:05.869793 25892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc420e871dc491eb7845c26499f96e2, State: Running, Role: LEADER
I20260619 02:49:05.870368 25887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:05.870296 25892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } }
I20260619 02:49:05.873878 25893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc420e871dc491eb7845c26499f96e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } } }
I20260619 02:49:05.874413 25893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:05.874276 25894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc420e871dc491eb7845c26499f96e2. Latest consensus state: current_term: 1 leader_uuid: "8bc420e871dc491eb7845c26499f96e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc420e871dc491eb7845c26499f96e2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36719 } } }
I20260619 02:49:05.874562 25894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:05.876137 25902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:05.881212 25902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:05.890084 25902 catalog_manager.cc:1347] Generated new cluster ID: 982ac61fe1c04e93a175f3dc36da505e
I20260619 02:49:05.890223 25902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:05.901261 25902 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:05.903163 25902 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:05.910606 25902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: Generated new TSK 0
I20260619 02:49:05.911616 25902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:05.919028 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--builtin_ntp_servers=127.7.246.84:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:06.247102 25911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:06.247430 25911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:06.247578 25911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:06.254845 25911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:06.254971 25911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:06.255131 25911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:06.264642 25911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:06.266036 25911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:06.267740 25911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:06.282019 25916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:06.282130 25917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:06.282099 25919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:06.283244 25911 server_base.cc:1034] running on GCE node
I20260619 02:49:06.283809 25911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:06.284762 25911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:06.286052 25911 hybrid_clock.cc:648] HybridClock initialized: now 1781837346286004 us; error 56 us; skew 500 ppm
I20260619 02:49:06.286409 25911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:06.288568 25911 webserver.cc:460] Webserver started at http://127.7.246.65:35625/ using document root <none> and password file <none>
I20260619 02:49:06.289268 25911 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:06.289402 25911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:06.289777 25911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:06.292577 25911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "24fc8665f1704c9fba69ac8056bb5227"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "e6d1beef3759ea07f3a13747dee73b55"
server_key_iv: "5c3dd7956360688db5e09b0a19c283cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.293316 25911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "24fc8665f1704c9fba69ac8056bb5227"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "e6d1beef3759ea07f3a13747dee73b55"
server_key_iv: "5c3dd7956360688db5e09b0a19c283cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.298254 25911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:49:06.301754 25925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.303032 25911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 02:49:06.303194 25911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "24fc8665f1704c9fba69ac8056bb5227"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "e6d1beef3759ea07f3a13747dee73b55"
server_key_iv: "5c3dd7956360688db5e09b0a19c283cb"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.303418 25911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:06.361773 25911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:06.362655 25911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:06.362948 25911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:06.363903 25911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:06.365360 25911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:06.365474 25911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.365644 25911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:06.365725 25911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.397951 25911 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36163
I20260619 02:49:06.398017 26037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36163 every 8 connection(s)
I20260619 02:49:06.399253 25911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:06.403215 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 25911
I20260619 02:49:06.403405 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:06.403999 8153 external_mini_cluster.cc:1428] Setting key ccfb94c51d73c02dd98b1d6df4cd117f
I20260619 02:49:06.406960 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--builtin_ntp_servers=127.7.246.84:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:06.413414 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:06.401739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:57113 (local address 127.7.246.126:36719)
0619 02:49:06.402178 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:06.402183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:06.403411 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 02:49:06.405479 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:06.405484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:06.405581 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:06.405717 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:06.407112 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.407923 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.410370 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.410929 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.411025 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:06.412617 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:06.412621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:06.412624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:06.412640 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:06.412678 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:06.412690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:06.412750 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:06.413019 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 02:49:06.413134 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":248,"threads_started":1}
I20260619 02:49:06.414795 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:06.402231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36719 (local address 127.7.246.65:57113)
0619 02:49:06.403210 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:06.403256 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:06.405210 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:06.405953 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:06.405964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:06.406073 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:06.406902 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:06.406938 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.408085 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.408095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:06.410193 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:06.410209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.411147 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.411156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:06.411492 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:06.412227 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:06.412233 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:06.412295 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:06.412419 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:06.412780 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:06.412793 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:06.412803 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:06.412979 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:49:06.413366 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":244,"threads_started":1}
I20260619 02:49:06.416216 26038 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36719
I20260619 02:49:06.416539 26038 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:06.417376 26038 heartbeater.cc:511] Master 127.7.246.126:36719 requested a full tablet report, sending...
I20260619 02:49:06.419584 25852 ts_manager.cc:194] Registered new tserver with Master: 24fc8665f1704c9fba69ac8056bb5227 (127.7.246.65:36163)
I20260619 02:49:06.421695 25852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.65:57113
W20260619 02:49:06.734855 26042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:06.735127 26042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:06.735216 26042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:06.742276 26042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:06.742375 26042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:06.742554 26042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:06.751451 26042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:06.752748 26042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:06.754530 26042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:06.767580 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:06.767513 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:06.767555 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:06.768756 26042 server_base.cc:1034] running on GCE node
I20260619 02:49:06.769475 26042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:06.770582 26042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:06.771847 26042 hybrid_clock.cc:648] HybridClock initialized: now 1781837346771785 us; error 49 us; skew 500 ppm
I20260619 02:49:06.772228 26042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:06.774797 26042 webserver.cc:460] Webserver started at http://127.7.246.66:44665/ using document root <none> and password file <none>
I20260619 02:49:06.775535 26042 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:06.775678 26042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:06.776021 26042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:06.779109 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "15e8ea9053be4fd58bd0583adfd7dd0e"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "0b55ffab604841c8cb89e96426807117"
server_key_iv: "27d8e476ad3d2881f3dcb6520cd30f25"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.779996 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "15e8ea9053be4fd58bd0583adfd7dd0e"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "0b55ffab604841c8cb89e96426807117"
server_key_iv: "27d8e476ad3d2881f3dcb6520cd30f25"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.785337 26042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:06.788971 26056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.790380 26042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:06.790598 26042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "15e8ea9053be4fd58bd0583adfd7dd0e"
format_stamp: "Formatted at 2026-06-19 02:49:06 on dist-test-slave-9tkz"
server_key: "0b55ffab604841c8cb89e96426807117"
server_key_iv: "27d8e476ad3d2881f3dcb6520cd30f25"
server_key_version: "encryptionkey@0"
I20260619 02:49:06.790829 26042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:06.811630 26042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:06.812479 26042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:06.812789 26042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:06.813896 26042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:06.815354 26042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:06.815481 26042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.815615 26042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:06.815702 26042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:06.850030 26042 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:37777
I20260619 02:49:06.850117 26168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:37777 every 8 connection(s)
I20260619 02:49:06.851410 26042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:06.854627 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26042
I20260619 02:49:06.854799 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:06.855250 8153 external_mini_cluster.cc:1428] Setting key 217fd5814a626be2e1a3c34e0caa5b3d
I20260619 02:49:06.858155 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--builtin_ntp_servers=127.7.246.84:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:06.864508 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:06.853892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:59593 (local address 127.7.246.126:36719)
0619 02:49:06.854134 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:06.854141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:06.855748 (+ 1607us) server_negotiation.cc:407] Connection header received
0619 02:49:06.857919 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:06.857924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:06.858011 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:06.858135 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:06.859321 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.860021 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.861605 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.862121 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.862261 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:06.863675 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:06.863680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:06.863683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:06.863702 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:06.863772 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:06.863784 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:06.863882 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:06.864111 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 02:49:06.864204 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:49:06.866019 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:06.854502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36719 (local address 127.7.246.66:59593)
0619 02:49:06.855580 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:06.855631 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:06.857658 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:06.858435 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:06.858449 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:06.858572 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:06.859143 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:06.859159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.860185 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.860190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:06.861438 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:06.861450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:06.862350 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:06.862357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:06.862586 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:06.863347 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:06.863350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:06.863390 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:06.863471 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:06.863882 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:06.863896 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:06.863906 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:06.864087 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 02:49:06.864497 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":371,"threads_started":1}
I20260619 02:49:06.867408 26169 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36719
I20260619 02:49:06.867725 26169 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:06.868467 26169 heartbeater.cc:511] Master 127.7.246.126:36719 requested a full tablet report, sending...
I20260619 02:49:06.869930 25852 ts_manager.cc:194] Registered new tserver with Master: 15e8ea9053be4fd58bd0583adfd7dd0e (127.7.246.66:37777)
I20260619 02:49:06.870812 25852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.66:59593
W20260619 02:49:07.182971 26172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:07.183298 26172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:07.183470 26172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:07.190964 26172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:07.191100 26172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:07.191280 26172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:07.200637 26172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:07.202019 26172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:07.203760 26172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:07.216907 26177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:07.216872 26178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:07.217031 26180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:07.217657 26172 server_base.cc:1034] running on GCE node
I20260619 02:49:07.218314 26172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:07.219267 26172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:07.220502 26172 hybrid_clock.cc:648] HybridClock initialized: now 1781837347220466 us; error 52 us; skew 500 ppm
I20260619 02:49:07.220832 26172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:07.223151 26172 webserver.cc:460] Webserver started at http://127.7.246.67:36033/ using document root <none> and password file <none>
I20260619 02:49:07.223842 26172 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:07.223973 26172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:07.224290 26172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:07.227161 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6"
format_stamp: "Formatted at 2026-06-19 02:49:07 on dist-test-slave-9tkz"
server_key: "69154511efce0dd1a12f4602b0713ba3"
server_key_iv: "18da15ec7ca702b191c8e2015eaa927b"
server_key_version: "encryptionkey@0"
I20260619 02:49:07.227945 26172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6"
format_stamp: "Formatted at 2026-06-19 02:49:07 on dist-test-slave-9tkz"
server_key: "69154511efce0dd1a12f4602b0713ba3"
server_key_iv: "18da15ec7ca702b191c8e2015eaa927b"
server_key_version: "encryptionkey@0"
I20260619 02:49:07.233026 26172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:07.236558 26186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:07.237911 26172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:07.238083 26172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6"
format_stamp: "Formatted at 2026-06-19 02:49:07 on dist-test-slave-9tkz"
server_key: "69154511efce0dd1a12f4602b0713ba3"
server_key_iv: "18da15ec7ca702b191c8e2015eaa927b"
server_key_version: "encryptionkey@0"
I20260619 02:49:07.238315 26172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:07.263378 26172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:07.264338 26172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:07.264802 26172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:07.265856 26172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:07.267410 26172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:07.267544 26172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:07.267683 26172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:07.267778 26172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:07.302131 26172 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39073
I20260619 02:49:07.302392 26298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39073 every 8 connection(s)
I20260619 02:49:07.303419 26172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:07.312892 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26172
I20260619 02:49:07.313079 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:07.313529 8153 external_mini_cluster.cc:1428] Setting key 433f6f3bc5e427fb8b056c289a5b1189
I20260619 02:49:07.315750 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.305979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:57229 (local address 127.7.246.126:36719)
0619 02:49:07.306150 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.306155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.307536 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 02:49:07.309309 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.309315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.309409 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.309519 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:07.310599 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.311337 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.312865 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.313381 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.313508 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.314889 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:07.314895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:07.314898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:07.314916 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:07.314950 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:07.314962 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:07.315047 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.315322 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.315420 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:49:07.316978 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.306453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:36719 (local address 127.7.246.67:57229)
0619 02:49:07.307372 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.307411 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.309036 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.309779 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.309786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.309878 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:07.310424 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.310440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.311491 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.311499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.312713 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.312724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.313640 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.313648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.313879 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.314522 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:07.314525 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:07.314564 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:07.314689 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:07.315064 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:07.315075 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:07.315083 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.315306 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.315734 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":266,"threads_started":1}
I20260619 02:49:07.318537 26299 heartbeater.cc:347] Connected to a master server at 127.7.246.126:36719
I20260619 02:49:07.318858 26299 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:07.319607 26299 heartbeater.cc:511] Master 127.7.246.126:36719 requested a full tablet report, sending...
I20260619 02:49:07.321004 25852 ts_manager.cc:194] Registered new tserver with Master: c69003cbb4fe479c8d0ef1bb8cc0b2b6 (127.7.246.67:39073)
I20260619 02:49:07.321985 25852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.67:57229
I20260619 02:49:07.329236 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:07.339089 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.333264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.7.246.126:36719)
0619 02:49:07.333573 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.333582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.333802 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:49:07.333969 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.333973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.334048 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.334131 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:07.334760 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.335483 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.336856 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.337239 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.337330 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.338419 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:07.338424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:07.338427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:07.338446 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:07.338478 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:07.338489 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:07.338558 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.338678 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.338748 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 02:49:07.342988 25852 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:07.350263 25852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:07.379024 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.363056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.7.246.67:39073)
0619 02:49:07.366648 (+ 3592us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.366674 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.367746 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 02:49:07.367846 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.367854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.368095 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.368303 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:07.369471 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.371436 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.374661 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.376677 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.376844 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.377207 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.378458 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.378627 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3350,"thread_start_us":3126,"threads_started":1}
I20260619 02:49:07.379134 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.363575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.7.246.66:37777)
0619 02:49:07.364305 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.364318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.365487 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 02:49:07.365632 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.365643 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.366602 (+ 959us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.366836 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:07.367696 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.370052 (+ 2356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.374485 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.376745 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.376905 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.377247 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.378306 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.378598 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":347,"threads_started":1}
I20260619 02:49:07.380164 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.362907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39073 (local address 127.0.0.1:45364)
0619 02:49:07.367469 (+ 4562us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.367483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.367622 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.368683 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.368691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.368725 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:07.369203 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.369218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.371633 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.371641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.374435 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.374451 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.377383 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.377393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.377750 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.378281 (+ 531us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.378380 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.379643 (+ 1263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4447,"mutex_wait_us":461,"thread_start_us":4249,"threads_started":1}
I20260619 02:49:07.380240 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.362887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37777 (local address 127.0.0.1:56036)
0619 02:49:07.364983 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.365040 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.365253 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.366866 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.366879 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.366930 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:07.367453 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.367475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.370303 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.370312 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.374164 (+ 3852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.374189 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.377498 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.377508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.377946 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.378015 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.379489 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.379731 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"mutex_wait_us":57,"thread_start_us":1603,"threads_started":1}
I20260619 02:49:07.384387 26234 tablet_service.cc:1467] Processing CreateTablet for tablet f07fe27f300c4f89870cd1bdef596926 (DEFAULT_TABLE table=test-table [id=e778de28b48e489089fb6f436a809204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:07.384382 26104 tablet_service.cc:1467] Processing CreateTablet for tablet f07fe27f300c4f89870cd1bdef596926 (DEFAULT_TABLE table=test-table [id=e778de28b48e489089fb6f436a809204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:07.385926 26234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07fe27f300c4f89870cd1bdef596926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:07.385922 26104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07fe27f300c4f89870cd1bdef596926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:07.387694 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.367616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49408)
0619 02:49:07.380107 (+ 12491us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.380128 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.380523 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.381221 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.381229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.381260 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:07.381641 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.381652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.383020 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.383029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.384555 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.384571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.386838 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.386845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.387038 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.387076 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.387212 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.387355 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12347,"thread_start_us":12192,"threads_started":1}
I20260619 02:49:07.387830 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.367898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.7.246.65:36163)
0619 02:49:07.378289 (+ 10391us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.378298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.380269 (+ 1971us) server_negotiation.cc:407] Connection header received
0619 02:49:07.380721 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.380733 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.381000 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.381200 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:07.381809 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.382870 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.384819 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.386594 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.386807 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.387062 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.387275 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.387438 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10243,"thread_start_us":10125,"threads_started":1}
I20260619 02:49:07.391484 25973 tablet_service.cc:1467] Processing CreateTablet for tablet f07fe27f300c4f89870cd1bdef596926 (DEFAULT_TABLE table=test-table [id=e778de28b48e489089fb6f436a809204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:07.392946 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07fe27f300c4f89870cd1bdef596926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:07.410202 26317 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Bootstrap starting.
I20260619 02:49:07.410914 26319 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: Bootstrap starting.
I20260619 02:49:07.411273 26318 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Bootstrap starting.
I20260619 02:49:07.413926 26319 tablet_bootstrap.cc:654] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:07.414044 26318 tablet_bootstrap.cc:654] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:07.415055 26319 log.cc:824] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:07.415315 26318 log.cc:824] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:07.416715 26317 tablet_bootstrap.cc:654] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:07.417958 26317 log.cc:824] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:07.418097 26318 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: No bootstrap required, opened a new log
I20260619 02:49:07.418253 26319 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: No bootstrap required, opened a new log
I20260619 02:49:07.418399 26318 ts_tablet_manager.cc:1397] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 02:49:07.418558 26319 ts_tablet_manager.cc:1397] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 02:49:07.421003 26317 tablet_bootstrap.cc:492] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: No bootstrap required, opened a new log
I20260619 02:49:07.421260 26317 ts_tablet_manager.cc:1397] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260619 02:49:07.424466 26038 heartbeater.cc:503] Master 127.7.246.126:36719 was elected leader, sending a full tablet report...
I20260619 02:49:07.427899 26318 raft_consensus.cc:348] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.428392 26318 raft_consensus.cc:374] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:07.428512 26318 raft_consensus.cc:729] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69003cbb4fe479c8d0ef1bb8cc0b2b6, State: Initialized, Role: FOLLOWER
I20260619 02:49:07.428989 26319 raft_consensus.cc:348] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.429425 26319 raft_consensus.cc:374] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:07.429287 26318 consensus_queue.cc:260] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.429530 26319 raft_consensus.cc:729] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fc8665f1704c9fba69ac8056bb5227, State: Initialized, Role: FOLLOWER
I20260619 02:49:07.430145 26319 consensus_queue.cc:260] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.430613 26299 heartbeater.cc:503] Master 127.7.246.126:36719 was elected leader, sending a full tablet report...
I20260619 02:49:07.431305 26318 ts_tablet_manager.cc:1428] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
I20260619 02:49:07.433935 26317 raft_consensus.cc:348] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.434425 26317 raft_consensus.cc:374] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:07.434571 26317 raft_consensus.cc:729] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e8ea9053be4fd58bd0583adfd7dd0e, State: Initialized, Role: FOLLOWER
I20260619 02:49:07.434966 26319 ts_tablet_manager.cc:1428] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260619 02:49:07.435312 26317 consensus_queue.cc:260] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.436412 26169 heartbeater.cc:503] Master 127.7.246.126:36719 was elected leader, sending a full tablet report...
I20260619 02:49:07.437027 26317 ts_tablet_manager.cc:1428] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
W20260619 02:49:07.555107 26300 tablet.cc:2369] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:07.561862 26324 raft_consensus.cc:482] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:07.562155 26324 raft_consensus.cc:504] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.564059 26324 leader_election.cc:288] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c69003cbb4fe479c8d0ef1bb8cc0b2b6 (127.7.246.67:39073), 15e8ea9053be4fd58bd0583adfd7dd0e (127.7.246.66:37777)
I20260619 02:49:07.570793 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.564341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:39381 (local address 127.7.246.67:39073)
0619 02:49:07.564537 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.564543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.565280 (+ 737us) server_negotiation.cc:407] Connection header received
0619 02:49:07.565498 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.565526 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.565630 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.565724 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:07.566329 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.567092 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.569103 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.569886 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.570051 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.570119 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.570438 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.570549 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:49:07.570835 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.564557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52273 (local address 127.7.246.66:37777)
0619 02:49:07.564811 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.564816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.565374 (+ 558us) server_negotiation.cc:407] Connection header received
0619 02:49:07.565610 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.565617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.565730 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.565818 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:07.566492 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.567297 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.569069 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.569851 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.569979 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.570048 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.570406 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.570530 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:49:07.570971 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.564239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39073 (local address 127.7.246.65:39381)
0619 02:49:07.565152 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.565178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.565345 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.565770 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.565777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.565806 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:07.566168 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.566177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.567212 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.567217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.568883 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.568894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.570091 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.570096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.570275 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.570297 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.570404 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.570560 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":125,"thread_start_us":501,"threads_started":1}
I20260619 02:49:07.571183 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.564502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:37777 (local address 127.7.246.65:52273)
0619 02:49:07.565252 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.565266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:07.565435 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:07.565902 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:07.565908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:07.565936 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:07.566318 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.566329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.567441 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.567444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.568869 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:07.568880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.570016 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.570022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:07.570184 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.570242 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:49:07.570358 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:49:07.570487 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":41,"thread_start_us":481,"threads_started":1}
I20260619 02:49:07.572043 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07fe27f300c4f89870cd1bdef596926" candidate_uuid: "24fc8665f1704c9fba69ac8056bb5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" is_pre_election: true
I20260619 02:49:07.572155 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07fe27f300c4f89870cd1bdef596926" candidate_uuid: "24fc8665f1704c9fba69ac8056bb5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" is_pre_election: true
I20260619 02:49:07.572582 26254 raft_consensus.cc:2454] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fc8665f1704c9fba69ac8056bb5227 in term 0.
I20260619 02:49:07.572662 26124 raft_consensus.cc:2454] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24fc8665f1704c9fba69ac8056bb5227 in term 0.
I20260619 02:49:07.573258 25926 leader_election.cc:302] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24fc8665f1704c9fba69ac8056bb5227, c69003cbb4fe479c8d0ef1bb8cc0b2b6; no voters:
I20260619 02:49:07.573673 26324 raft_consensus.cc:2781] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:07.573809 26324 raft_consensus.cc:482] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:07.573912 26324 raft_consensus.cc:3037] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:07.576107 26324 raft_consensus.cc:504] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.576747 26324 leader_election.cc:288] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [CANDIDATE]: Term 1 election: Requested vote from peers c69003cbb4fe479c8d0ef1bb8cc0b2b6 (127.7.246.67:39073), 15e8ea9053be4fd58bd0583adfd7dd0e (127.7.246.66:37777)
I20260619 02:49:07.577158 26254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07fe27f300c4f89870cd1bdef596926" candidate_uuid: "24fc8665f1704c9fba69ac8056bb5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6"
I20260619 02:49:07.577203 26124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07fe27f300c4f89870cd1bdef596926" candidate_uuid: "24fc8665f1704c9fba69ac8056bb5227" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e"
I20260619 02:49:07.577351 26254 raft_consensus.cc:3037] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:07.577368 26124 raft_consensus.cc:3037] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:07.579555 26124 raft_consensus.cc:2454] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fc8665f1704c9fba69ac8056bb5227 in term 1.
I20260619 02:49:07.579595 26254 raft_consensus.cc:2454] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24fc8665f1704c9fba69ac8056bb5227 in term 1.
I20260619 02:49:07.579964 25927 leader_election.cc:302] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e8ea9053be4fd58bd0583adfd7dd0e, 24fc8665f1704c9fba69ac8056bb5227; no voters:
I20260619 02:49:07.580269 26324 raft_consensus.cc:2781] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:07.580724 26324 raft_consensus.cc:686] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 1 LEADER]: Becoming Leader. State: Replica: 24fc8665f1704c9fba69ac8056bb5227, State: Running, Role: LEADER
I20260619 02:49:07.581315 26324 consensus_queue.cc:237] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } }
I20260619 02:49:07.585752 25851 catalog_manager.cc:5495] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24fc8665f1704c9fba69ac8056bb5227 (127.7.246.65). New cstate: current_term: 1 leader_uuid: "24fc8665f1704c9fba69ac8056bb5227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fc8665f1704c9fba69ac8056bb5227" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36163 } health_report { overall_health: HEALTHY } } }
W20260619 02:49:07.603387 26170 tablet.cc:2369] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:07.611733 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.606626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.7.246.65:36163)
0619 02:49:07.606794 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.606799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.606844 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:49:07.606971 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.606974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.607054 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.607171 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:07.607579 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.608338 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.609757 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.610168 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.610261 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.610652 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:07.611115 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:07.611219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.611317 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.611399 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:49:07.619839 26254 raft_consensus.cc:1261] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 1 FOLLOWER]: Refusing update from remote peer 24fc8665f1704c9fba69ac8056bb5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:07.619863 26124 raft_consensus.cc:1261] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 1 FOLLOWER]: Refusing update from remote peer 24fc8665f1704c9fba69ac8056bb5227: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:07.620770 26328 consensus_queue.cc:1035] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69003cbb4fe479c8d0ef1bb8cc0b2b6" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:07.621148 26324 consensus_queue.cc:1035] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e8ea9053be4fd58bd0583adfd7dd0e" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:07.633281 26331 mvcc.cc:205] Tried to move back new op lower bound from 7298405775840063488 to 7298405775697035264. Current Snapshot: MvccSnapshot[applied={T|T < 7298405775840063488}]
I20260619 02:49:07.636000 26333 mvcc.cc:205] Tried to move back new op lower bound from 7298405775840063488 to 7298405775697035264. Current Snapshot: MvccSnapshot[applied={T|T < 7298405775840063488}]
I20260619 02:49:07.645236 25852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:07.645494 25852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:07.648945 25852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bc420e871dc491eb7845c26499f96e2: Sending DeleteTablet for 3 replicas of tablet f07fe27f300c4f89870cd1bdef596926
I20260619 02:49:07.650043 26234 tablet_service.cc:1514] Processing DeleteTablet for tablet f07fe27f300c4f89870cd1bdef596926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:07 UTC) from {username='slave'} at 127.0.0.1:45364
I20260619 02:49:07.650086 26104 tablet_service.cc:1514] Processing DeleteTablet for tablet f07fe27f300c4f89870cd1bdef596926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:07 UTC) from {username='slave'} at 127.0.0.1:56036
I20260619 02:49:07.651188 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25911
I20260619 02:49:07.651291 25973 tablet_service.cc:1514] Processing DeleteTablet for tablet f07fe27f300c4f89870cd1bdef596926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:07 UTC) from {username='slave'} at 127.0.0.1:49408
I20260619 02:49:07.671582 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.653677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.7.246.65:36163)
0619 02:49:07.653904 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.653913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.662586 (+ 8673us) server_negotiation.cc:407] Connection header received
0619 02:49:07.663059 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.663065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.663166 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.663260 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:07.663953 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.665204 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.667282 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.667955 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.668055 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.669677 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:07.669688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:07.669708 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:07.670595 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:07.670662 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:07.670688 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:07.670775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.671052 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.671173 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:49:07.672778 26033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:07.674089 26341 tablet_replica.cc:330] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: stopping tablet replica
I20260619 02:49:07.674666 26341 raft_consensus.cc:2229] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:07.675351 26341 raft_consensus.cc:2258] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:07.676810 26340 tablet_replica.cc:330] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: stopping tablet replica
I20260619 02:49:07.677361 26340 raft_consensus.cc:2229] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:07.678009 26340 raft_consensus.cc:2258] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:07.678594 26341 ts_tablet_manager.cc:1905] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:07.680100 26340 ts_tablet_manager.cc:1905] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:07.684566 26340 ts_tablet_manager.cc:1918] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:07.684746 26340 log.cc:1196] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/f07fe27f300c4f89870cd1bdef596926
I20260619 02:49:07.685205 26340 ts_tablet_manager.cc:1939] T f07fe27f300c4f89870cd1bdef596926 P 15e8ea9053be4fd58bd0583adfd7dd0e: Deleting consensus metadata
I20260619 02:49:07.685881 26341 ts_tablet_manager.cc:1918] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:07.686059 26341 log.cc:1196] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/f07fe27f300c4f89870cd1bdef596926
I20260619 02:49:07.686563 26341 ts_tablet_manager.cc:1939] T f07fe27f300c4f89870cd1bdef596926 P c69003cbb4fe479c8d0ef1bb8cc0b2b6: Deleting consensus metadata
I20260619 02:49:07.686875 25837 catalog_manager.cc:4841] TS 15e8ea9053be4fd58bd0583adfd7dd0e (127.7.246.66:37777): tablet f07fe27f300c4f89870cd1bdef596926 (table test-table [id=e778de28b48e489089fb6f436a809204]) successfully deleted
I20260619 02:49:07.688444 25836 catalog_manager.cc:4841] TS c69003cbb4fe479c8d0ef1bb8cc0b2b6 (127.7.246.67:39073): tablet f07fe27f300c4f89870cd1bdef596926 (table test-table [id=e778de28b48e489089fb6f436a809204]) successfully deleted
I20260619 02:49:07.775028 26342 tablet_replica.cc:330] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227: stopping tablet replica
I20260619 02:49:07.775584 26342 raft_consensus.cc:2229] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:07.776330 26342 raft_consensus.cc:2258] T f07fe27f300c4f89870cd1bdef596926 P 24fc8665f1704c9fba69ac8056bb5227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:07.777298 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25911
I20260619 02:49:07.795051 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26042
W20260619 02:49:07.794981 25836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:36163 (error 108)
I20260619 02:49:07.796958 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.796508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49442)
0619 02:49:07.796693 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.796787 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:49:07.797228 25836 catalog_manager.cc:4584] TS 24fc8665f1704c9fba69ac8056bb5227 (127.7.246.65:36163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f07fe27f300c4f89870cd1bdef596926: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
I20260619 02:49:07.801910 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.795931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.7.246.66:37777)
0619 02:49:07.796130 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.796136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.796157 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:49:07.796306 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.796312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.796417 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.796498 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:07.796928 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.797833 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.799086 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.799454 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.799539 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.800511 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:07.800518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:07.800531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:07.801084 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:07.801126 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:07.801143 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:07.801228 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.801336 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.801412 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:49:07.802551 26164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:07.850790 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.850298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49454)
0619 02:49:07.850518 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.850616 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 02:49:07.851868 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.851505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49464)
0619 02:49:07.851654 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.851744 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 02:49:07.905516 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26042
I20260619 02:49:07.912492 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.912068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49466)
0619 02:49:07.912272 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.912353 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:49:07.913650 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.913276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49468)
0619 02:49:07.913413 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.913479 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:49:07.923300 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26172
I20260619 02:49:07.930066 26313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.924061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.7.246.67:39073)
0619 02:49:07.924236 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:07.924240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:07.924260 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:49:07.924430 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:07.924434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:07.924524 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:07.924604 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:07.925036 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.925855 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.927178 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:07.927581 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:07.927666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:07.928657 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:07.928665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:07.928679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:07.929220 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:07.929264 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:07.929286 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:07.929379 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:07.929645 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 02:49:07.929731 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:49:07.930755 26294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:07.994441 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.993951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49482)
0619 02:49:07.994163 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.994265 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:49:07.995488 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:07.995102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36163 (local address 127.0.0.1:49492)
0619 02:49:07.995307 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:07.995377 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 02:49:08.018304 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26172
I20260619 02:49:08.034584 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 25821
I20260619 02:49:08.035573 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:08.143175 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 25821
2026-06-19T02:49:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2819 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:08 dist-test-slave-9tkz krb5kdc[26351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:08 dist-test-slave-9tkz krb5kdc[26351](info): set up 2 sockets
Jun 19 02:49:08 dist-test-slave-9tkz krb5kdc[26351](info): commencing operation
krb5kdc: starting...
W20260619 02:49:09.245859 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:09 dist-test-slave-9tkz krb5kdc[26351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:09.481218 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:42709
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:09.809242 26364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:09.809528 26364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:09.809774 26364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:09.817094 26364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:09.817209 26364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:09.817327 26364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:09.817430 26364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:09.817533 26364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:09.828322 26364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:42709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:09.829748 26364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:09.831089 26364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:09.842813 26369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:09.842873 26370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:09.843099 26372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:09.844055 26364 server_base.cc:1034] running on GCE node
I20260619 02:49:09.844771 26364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:09.846202 26364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:09.847429 26364 hybrid_clock.cc:648] HybridClock initialized: now 1781837349847393 us; error 48 us; skew 500 ppm
I20260619 02:49:09.847759 26364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:09 dist-test-slave-9tkz krb5kdc[26351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:09.850750 26364 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:09.852202 26364 webserver.cc:460] Webserver started at http://127.7.246.126:38249/ using document root <none> and password file <none>
I20260619 02:49:09.852856 26364 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:09.853008 26364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:09.853295 26364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:09.856187 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "c0605fca01464b0899e15b159e5f8760"
format_stamp: "Formatted at 2026-06-19 02:49:09 on dist-test-slave-9tkz"
server_key: "e64e53fea059a4dbbefaf3d4f4028fdb"
server_key_iv: "799bc53455e56451960c4c8fe7fc79bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:09.856892 26364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "c0605fca01464b0899e15b159e5f8760"
format_stamp: "Formatted at 2026-06-19 02:49:09 on dist-test-slave-9tkz"
server_key: "e64e53fea059a4dbbefaf3d4f4028fdb"
server_key_iv: "799bc53455e56451960c4c8fe7fc79bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:09.861867 26364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:09.865070 26379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:09.866376 26364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:09.866552 26364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "c0605fca01464b0899e15b159e5f8760"
format_stamp: "Formatted at 2026-06-19 02:49:09 on dist-test-slave-9tkz"
server_key: "e64e53fea059a4dbbefaf3d4f4028fdb"
server_key_iv: "799bc53455e56451960c4c8fe7fc79bf"
server_key_version: "encryptionkey@0"
I20260619 02:49:09.866736 26364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:09.881438 26364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:09.885411 26364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:09.885701 26364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:09.904134 26364 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:42709
I20260619 02:49:09.904178 26430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:42709 every 8 connection(s)
I20260619 02:49:09.905526 26364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:09.909068 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:09.914219 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26364
I20260619 02:49:09.914398 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:09.914912 8153 external_mini_cluster.cc:1428] Setting key cc6479d48a738ef194d0d9fede28a5f1
I20260619 02:49:09.920156 26431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: Bootstrap starting.
I20260619 02:49:09.923834 26431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:09.925107 26431 log.cc:824] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:09.928591 26431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: No bootstrap required, opened a new log
Jun 19 02:49:09 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:09.938158 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:09.916674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.7.246.126:42709)
0619 02:49:09.917502 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:09.917517 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:09.917606 (+ 89us) server_negotiation.cc:407] Connection header received
0619 02:49:09.919315 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:09.919337 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:09.919675 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:09.919988 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:09.921673 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:09.922944 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:09.924875 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:09.925753 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:09.925970 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:09.930812 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:09.930821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:09.930834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:09.930878 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:09.933968 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:09.934779 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:09.934785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:09.934791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:09.934951 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:09.935413 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:09.935422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:09.935425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:09.936037 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:09.936252 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:09.936732 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 02:49:09.937067 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":425,"threads_started":1}
I20260619 02:49:09.938140 26431 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } }
I20260619 02:49:09.938618 26431 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:09.938747 26431 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0605fca01464b0899e15b159e5f8760, State: Initialized, Role: FOLLOWER
I20260619 02:49:09.939356 26431 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } }
I20260619 02:49:09.939616 26431 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:09.939779 26431 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:09.939951 26431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:09.942435 26431 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } }
I20260619 02:49:09.942907 26431 leader_election.cc:302] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0605fca01464b0899e15b159e5f8760; no voters:
I20260619 02:49:09.943454 26431 leader_election.cc:288] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:09.943542 26436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:09.944334 26436 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [term 1 LEADER]: Becoming Leader. State: Replica: c0605fca01464b0899e15b159e5f8760, State: Running, Role: LEADER
I20260619 02:49:09.945250 26436 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } }
I20260619 02:49:09.945700 26431 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:09.947939 26437 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0605fca01464b0899e15b159e5f8760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } } }
I20260619 02:49:09.948140 26438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0605fca01464b0899e15b159e5f8760. Latest consensus state: current_term: 1 leader_uuid: "c0605fca01464b0899e15b159e5f8760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0605fca01464b0899e15b159e5f8760" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 42709 } } }
I20260619 02:49:09.948366 26437 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:09.948383 26438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:09.950028 26445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:09.954836 26445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:09.963835 26445 catalog_manager.cc:1347] Generated new cluster ID: 60378e2d654748e8b4533f5953788e90
I20260619 02:49:09.963949 26445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:09.975523 26445 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:09.976750 26445 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:09.984702 26445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: Generated new TSK 0
I20260619 02:49:09.985502 26445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:49:10.052681 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:10.386256 26457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:10.386582 26457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:10.386744 26457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:10.393944 26457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:10.394099 26457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:10.394281 26457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:10.403759 26457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:10.405095 26457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:10.406419 26457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:10.419421 26463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:10.419461 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:10.419374 26462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:10.420209 26457 server_base.cc:1034] running on GCE node
I20260619 02:49:10.420771 26457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:10.421732 26457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:10.422928 26457 hybrid_clock.cc:648] HybridClock initialized: now 1781837350422890 us; error 29 us; skew 500 ppm
I20260619 02:49:10.423183 26457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:10 dist-test-slave-9tkz krb5kdc[26351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837350, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:10.430253 26457 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:49:10.431672 26457 webserver.cc:460] Webserver started at http://127.7.246.65:42523/ using document root <none> and password file <none>
I20260619 02:49:10.432354 26457 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:10.432482 26457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:10.432798 26457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:10.435691 26457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "2cc1060952314479af180c184836cc24"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "78a3b6702284c4e2a89c9c14bd6cc2b7"
server_key_iv: "f5c5a2afbe02551bdf67f413e368d3a4"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.436443 26457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "2cc1060952314479af180c184836cc24"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "78a3b6702284c4e2a89c9c14bd6cc2b7"
server_key_iv: "f5c5a2afbe02551bdf67f413e368d3a4"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.441476 26457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:10.444746 26472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:10.446225 26457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:10.446405 26457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "2cc1060952314479af180c184836cc24"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "78a3b6702284c4e2a89c9c14bd6cc2b7"
server_key_iv: "f5c5a2afbe02551bdf67f413e368d3a4"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.446605 26457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:10.467780 26457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:10.472150 26457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:10.472438 26457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:10.473484 26457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:10.474845 26457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:10.474927 26457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:10.475088 26457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:10.475152 26457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:10.506949 26457 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:37837
I20260619 02:49:10.507040 26584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:37837 every 8 connection(s)
I20260619 02:49:10.508138 26457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:10.509418 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26457
I20260619 02:49:10.509629 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:10.510084 8153 external_mini_cluster.cc:1428] Setting key 52899c5a08aeeec882b6b63e9746e89d
Jun 19 02:49:10 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837350, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:10.534510 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:10.511343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42709 (local address 127.7.246.65:52821)
0619 02:49:10.512134 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:10.512185 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:10.514040 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:10.514773 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:10.514781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:10.515272 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:10.515822 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:10.515837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:10.516847 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:10.516859 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:10.518170 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:10.518181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:10.518964 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:10.518974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:10.519211 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:10.520450 (+ 1239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:10.520482 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:10.524164 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:10.529340 (+ 5176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:10.529353 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:10.529373 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:10.530040 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:10.530505 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:10.530513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:10.530520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:10.530779 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:10.531586 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:10.531597 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:10.532054 (+ 457us) client_negotiation.cc:769] Sending connection context
0619 02:49:10.532357 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 02:49:10.532943 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":338,"threads_started":1}
I20260619 02:49:10.535813 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:10.510773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52821 (local address 127.7.246.126:42709)
0619 02:49:10.511228 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:10.511234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:10.512291 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 02:49:10.514322 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:10.514327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:10.514425 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:10.514542 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:10.515992 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:10.516693 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:10.518321 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:10.518743 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:10.518888 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:10.524681 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:10.524690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:10.524698 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:10.524739 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:10.529138 (+ 4399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:10.530189 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:10.530195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:10.530198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:10.530363 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:10.530924 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:10.530931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:10.530945 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:10.531341 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:10.531494 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:10.535083 (+ 3589us) server_negotiation.cc:299] Negotiation successful
0619 02:49:10.535359 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260619 02:49:10.537046 26585 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42709
I20260619 02:49:10.537441 26585 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:10.538509 26585 heartbeater.cc:511] Master 127.7.246.126:42709 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
I20260619 02:49:10.540874 26396 ts_manager.cc:194] Registered new tserver with Master: 2cc1060952314479af180c184836cc24 (127.7.246.65:37837)
I20260619 02:49:10.542788 26396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:52821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:49:10.580861 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:10.904769 26591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:10.905136 26591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:10.905328 26591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:10.917279 26591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:10.917429 26591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:10.917682 26591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:10.931216 26591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:10.932406 26591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:10.933765 26591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:10.946969 26596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:10.947046 26597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:10.947069 26599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:10.948290 26591 server_base.cc:1034] running on GCE node
I20260619 02:49:10.948844 26591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:10.949934 26591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:10.951213 26591 hybrid_clock.cc:648] HybridClock initialized: now 1781837350951112 us; error 78 us; skew 500 ppm
I20260619 02:49:10.951550 26591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:10 dist-test-slave-9tkz krb5kdc[26351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837350, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:10.957195 26591 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:49:10.958914 26591 webserver.cc:460] Webserver started at http://127.7.246.66:40837/ using document root <none> and password file <none>
I20260619 02:49:10.959645 26591 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:10.959779 26591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:10.960108 26591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:10.963153 26591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "27a01d61ace24654bd59d33e06a1b747"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "6290c5a1ac1211f5adb8eb3a5fbfa0e4"
server_key_iv: "d9cab6beafc337f976423a8ed25cf099"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.964038 26591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "27a01d61ace24654bd59d33e06a1b747"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "6290c5a1ac1211f5adb8eb3a5fbfa0e4"
server_key_iv: "d9cab6beafc337f976423a8ed25cf099"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.969208 26591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:10.972994 26606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:10.974570 26591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:10.974704 26591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "27a01d61ace24654bd59d33e06a1b747"
format_stamp: "Formatted at 2026-06-19 02:49:10 on dist-test-slave-9tkz"
server_key: "6290c5a1ac1211f5adb8eb3a5fbfa0e4"
server_key_iv: "d9cab6beafc337f976423a8ed25cf099"
server_key_version: "encryptionkey@0"
I20260619 02:49:10.974951 26591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:10.989131 26591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:10.993539 26591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:10.993886 26591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:10.994954 26591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:10.996403 26591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:10.996474 26591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:10.996593 26591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:10.996658 26591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:11.031229 26591 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:43439
I20260619 02:49:11.031283 26718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:43439 every 8 connection(s)
I20260619 02:49:11.032594 26591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:11.035458 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26591
I20260619 02:49:11.035635 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:11.036080 8153 external_mini_cluster.cc:1428] Setting key 48baef8b86383bdf8792c11075958ace
Jun 19 02:49:11 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837350, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:11.053380 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.035215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:60367 (local address 127.7.246.126:42709)
0619 02:49:11.035461 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.035467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.036818 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 02:49:11.038856 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.038861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.038945 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.039057 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:11.040558 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.041251 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.042836 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.043268 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.043417 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.047280 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:11.047289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:11.047295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:11.047339 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:11.050245 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.051053 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.051064 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.051066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.051187 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.051684 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.051688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.051692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.051955 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:11.052117 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.052784 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.053016 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 02:49:11.054538 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.035835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42709 (local address 127.7.246.66:60367)
0619 02:49:11.036660 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:11.036710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:11.038597 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:11.039299 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:11.039307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:11.039821 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:11.040394 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.040412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.041405 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.041410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.042692 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.042704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.043508 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.043516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.043750 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.044454 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:11.044476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:11.047006 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:11.050408 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:11.050416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:11.050432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:11.050907 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:11.051296 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:11.051301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:11.051305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:11.051570 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:11.052080 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:11.052090 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:11.052442 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 02:49:11.052698 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 02:49:11.053245 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":351,"threads_started":1}
I20260619 02:49:11.056062 26719 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42709
I20260619 02:49:11.056367 26719 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:11.057080 26719 heartbeater.cc:511] Master 127.7.246.126:42709 requested a full tablet report, sending...
I20260619 02:49:11.058686 26396 ts_manager.cc:194] Registered new tserver with Master: 27a01d61ace24654bd59d33e06a1b747 (127.7.246.66:43439)
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
I20260619 02:49:11.059824 26396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:60367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:49:11.090893 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:11.415879 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:11.416160 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:11.416366 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:11.423688 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:11.423797 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:11.423974 26724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:11.433972 26724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:42709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:11.435266 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:11.436589 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:11.450156 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:11.450179 26732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:11.450434 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:11.451610 26724 server_base.cc:1034] running on GCE node
I20260619 02:49:11.452312 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:11.453256 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:11.454466 26724 hybrid_clock.cc:648] HybridClock initialized: now 1781837351454396 us; error 63 us; skew 500 ppm
I20260619 02:49:11.454838 26724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:11 dist-test-slave-9tkz krb5kdc[26351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:11.457998 26724 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:49:11.459478 26724 webserver.cc:460] Webserver started at http://127.7.246.67:40857/ using document root <none> and password file <none>
I20260619 02:49:11.460130 26724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:11.460232 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:11.460484 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:11.463366 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "19b667acaf5842b590581170fd1f0808"
format_stamp: "Formatted at 2026-06-19 02:49:11 on dist-test-slave-9tkz"
server_key: "8f3eb3ffe7e7e0c54355b1f8ad34c2b4"
server_key_iv: "1b65f81887650acc5a75906af41d49c0"
server_key_version: "encryptionkey@0"
I20260619 02:49:11.464112 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "19b667acaf5842b590581170fd1f0808"
format_stamp: "Formatted at 2026-06-19 02:49:11 on dist-test-slave-9tkz"
server_key: "8f3eb3ffe7e7e0c54355b1f8ad34c2b4"
server_key_iv: "1b65f81887650acc5a75906af41d49c0"
server_key_version: "encryptionkey@0"
I20260619 02:49:11.469201 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:11.472754 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:11.474133 26724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:11.474262 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "19b667acaf5842b590581170fd1f0808"
format_stamp: "Formatted at 2026-06-19 02:49:11 on dist-test-slave-9tkz"
server_key: "8f3eb3ffe7e7e0c54355b1f8ad34c2b4"
server_key_iv: "1b65f81887650acc5a75906af41d49c0"
server_key_version: "encryptionkey@0"
I20260619 02:49:11.474412 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:11.500885 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:11.504660 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:11.504932 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:11.505883 26724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:11.507493 26724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:11.507593 26724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:11.507746 26724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:11.507813 26724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:11.542320 26724 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:40141
I20260619 02:49:11.542362 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:40141 every 8 connection(s)
I20260619 02:49:11.543697 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:11.544772 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26724
I20260619 02:49:11.544956 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:11.545571 8153 external_mini_cluster.cc:1428] Setting key a51499d5cdcdcaef697f9bd2871ee89e
I20260619 02:49:11.546743 26585 heartbeater.cc:503] Master 127.7.246.126:42709 was elected leader, sending a full tablet report...
Jun 19 02:49:11 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837351, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:11.565285 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.546538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:39655 (local address 127.7.246.126:42709)
0619 02:49:11.546817 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.546825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.548421 (+ 1596us) server_negotiation.cc:407] Connection header received
0619 02:49:11.550648 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.550653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.550734 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.550879 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:11.552355 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.553106 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.554717 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.555128 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.555217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.558638 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:11.558643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:11.558646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:11.558677 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:11.562020 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.562791 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.562797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.562801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.562950 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.563423 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.563430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.563439 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.563731 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:11.563906 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.564612 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.564863 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 02:49:11.566325 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.547275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:42709 (local address 127.7.246.67:39655)
0619 02:49:11.548246 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:11.548293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:11.550354 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:11.551088 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:11.551095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:11.551615 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:11.552208 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.552227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.553252 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.553256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.554587 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.554600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.555319 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.555326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.555532 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.556155 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:11.556179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:11.558452 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:11.562178 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:11.562187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:11.562211 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:11.562646 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:11.563072 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:11.563078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:11.563082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:11.563291 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:11.563875 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:11.563883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:11.564233 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 02:49:11.564512 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 02:49:11.565042 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":404,"threads_started":1}
I20260619 02:49:11.567651 26852 heartbeater.cc:347] Connected to a master server at 127.7.246.126:42709
I20260619 02:49:11.567962 26852 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:11.568665 26852 heartbeater.cc:511] Master 127.7.246.126:42709 requested a full tablet report, sending...
I20260619 02:49:11.570019 26395 ts_manager.cc:194] Registered new tserver with Master: 19b667acaf5842b590581170fd1f0808 (127.7.246.67:40141)
I20260619 02:49:11.570962 26395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:39655
I20260619 02:49:11.573437 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:11.590900 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.577427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.7.246.126:42709)
0619 02:49:11.577649 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.577657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.578115 (+ 458us) server_negotiation.cc:407] Connection header received
0619 02:49:11.578236 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.578239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.578294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.578427 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:11.579900 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.580600 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.582243 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.582677 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.582772 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.585609 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:11.585614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:11.585617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:11.585648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:11.588262 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.589005 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.589009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.589012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.589132 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:11.589503 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:11.589506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:11.589508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:11.589858 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:11.590047 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.590372 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.590504 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:49:11.595237 26395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:11.602258 26395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:11.627079 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.614628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:40141 (local address 127.0.0.1:47266)
0619 02:49:11.616074 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:11.616118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:11.616270 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:11.620031 (+ 3761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:11.620035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:11.620060 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:11.620460 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.620475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.622479 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.622485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.624214 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.624226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.626071 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.626080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.626257 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.626304 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:49:11.626417 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:49:11.626634 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":1304,"threads_started":1}
I20260619 02:49:11.627401 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.615534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.7.246.65:37837)
0619 02:49:11.616914 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.616922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.618173 (+ 1251us) server_negotiation.cc:407] Connection header received
0619 02:49:11.618313 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.618325 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.618682 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.618913 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:11.619980 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.621380 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.624596 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.626021 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.626151 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.626390 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.626915 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.627078 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1033,"threads_started":1}
I20260619 02:49:11.627655 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.616728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:43439 (local address 127.0.0.1:58114)
0619 02:49:11.619391 (+ 2663us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:11.619409 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:11.619528 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:11.620346 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:11.620351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:11.620383 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:11.620770 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.620783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.622712 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.622718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.624796 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.624805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.626852 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.626860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.627022 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.627060 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:49:11.627146 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:49:11.627275 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2497,"mutex_wait_us":1,"thread_start_us":703,"threads_started":1}
I20260619 02:49:11.627897 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.615243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:37837 (local address 127.0.0.1:47144)
0619 02:49:11.617835 (+ 2592us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:11.617860 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:11.618017 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:11.619263 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:11.619273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:11.619308 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:11.619801 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.619815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.621598 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.621602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.624432 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:11.624448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.626535 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.626541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:11.626708 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.626740 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:49:11.626850 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:49:11.626963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2394,"thread_start_us":3821,"threads_started":1}
I20260619 02:49:11.628247 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.614868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.7.246.67:40141)
0619 02:49:11.617439 (+ 2571us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.617447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.617481 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:49:11.619667 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.619673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.619833 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.620036 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:11.620660 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.622324 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.624371 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.625828 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.627194 (+ 1366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.627421 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.627679 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.627875 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":2208,"threads_started":1}
I20260619 02:49:11.628382 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:11.617199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.7.246.66:43439)
0619 02:49:11.619680 (+ 2481us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:11.619689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:11.619730 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:11.619838 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:11.619846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:11.620088 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:11.620301 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:11.620961 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.622578 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.624932 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:11.626364 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:11.626503 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:11.626687 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:11.627884 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0619 02:49:11.628055 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2317,"thread_start_us":2176,"threads_started":1}
I20260619 02:49:11.631759 26787 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc (DEFAULT_TABLE table=test-table [id=ba214682924749c381e0bc92d337a61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:11.631932 26654 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc (DEFAULT_TABLE table=test-table [id=ba214682924749c381e0bc92d337a61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:11.631932 26520 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc (DEFAULT_TABLE table=test-table [id=ba214682924749c381e0bc92d337a61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:11.633040 26787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bb1a546414339ad1201bc5ecba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:11.633302 26520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bb1a546414339ad1201bc5ecba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:11.633309 26654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2bb1a546414339ad1201bc5ecba0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:11.646512 26870 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Bootstrap starting.
I20260619 02:49:11.648375 26872 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Bootstrap starting.
I20260619 02:49:11.650179 26871 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Bootstrap starting.
I20260619 02:49:11.650933 26870 tablet_bootstrap.cc:654] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:11.652258 26870 log.cc:824] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:11.652796 26872 tablet_bootstrap.cc:654] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:11.653154 26871 tablet_bootstrap.cc:654] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:11.654100 26872 log.cc:824] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:11.654388 26871 log.cc:824] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:11.655884 26870 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: No bootstrap required, opened a new log
I20260619 02:49:11.656188 26870 ts_tablet_manager.cc:1397] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260619 02:49:11.657315 26872 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: No bootstrap required, opened a new log
I20260619 02:49:11.657640 26872 ts_tablet_manager.cc:1397] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 02:49:11.658442 26871 tablet_bootstrap.cc:492] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: No bootstrap required, opened a new log
I20260619 02:49:11.658705 26871 ts_tablet_manager.cc:1397] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 02:49:11.664662 26872 raft_consensus.cc:348] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.665139 26872 raft_consensus.cc:374] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:11.665239 26872 raft_consensus.cc:729] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a01d61ace24654bd59d33e06a1b747, State: Initialized, Role: FOLLOWER
I20260619 02:49:11.665900 26872 consensus_queue.cc:260] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.667273 26719 heartbeater.cc:503] Master 127.7.246.126:42709 was elected leader, sending a full tablet report...
I20260619 02:49:11.666934 26870 raft_consensus.cc:348] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.667518 26870 raft_consensus.cc:374] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:11.667623 26870 raft_consensus.cc:729] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b667acaf5842b590581170fd1f0808, State: Initialized, Role: FOLLOWER
I20260619 02:49:11.668253 26872 ts_tablet_manager.cc:1428] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 02:49:11.668422 26870 consensus_queue.cc:260] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.669602 26871 raft_consensus.cc:348] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.669934 26852 heartbeater.cc:503] Master 127.7.246.126:42709 was elected leader, sending a full tablet report...
I20260619 02:49:11.670053 26871 raft_consensus.cc:374] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:11.670148 26871 raft_consensus.cc:729] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc1060952314479af180c184836cc24, State: Initialized, Role: FOLLOWER
I20260619 02:49:11.670682 26870 ts_tablet_manager.cc:1428] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 02:49:11.670925 26871 consensus_queue.cc:260] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:11.672724 26871 ts_tablet_manager.cc:1428] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260619 02:49:11.760444 26586 tablet.cc:2369] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:11.784699 26720 tablet.cc:2369] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:11.795266 26853 tablet.cc:2369] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:12.045903 26876 raft_consensus.cc:482] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:12.046180 26876 raft_consensus.cc:504] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:12.047855 26876 leader_election.cc:288] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b667acaf5842b590581170fd1f0808 (127.7.246.67:40141), 2cc1060952314479af180c184836cc24 (127.7.246.65:37837)
I20260619 02:49:12.054874 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.048249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:44217 (local address 127.7.246.67:40141)
0619 02:49:12.048648 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.048655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.049091 (+ 436us) server_negotiation.cc:407] Connection header received
0619 02:49:12.049327 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.049332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.049406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.049526 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:12.050247 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.051206 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.053027 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.053883 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.054024 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.054115 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.054487 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.054614 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260619 02:49:12.054884 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.048455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:34511 (local address 127.7.246.65:37837)
0619 02:49:12.048632 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.048638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.049342 (+ 704us) server_negotiation.cc:407] Connection header received
0619 02:49:12.049552 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.049582 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.049677 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.049815 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:12.050524 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.051358 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.053055 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.053911 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.054101 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.054177 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.054487 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.054616 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:49:12.055109 26879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.048066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:40141 (local address 127.7.246.66:44217)
0619 02:49:12.048951 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:12.048981 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:12.049154 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:12.049579 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:12.049584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:12.049623 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:12.050105 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:12.050116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.051339 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.051345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:12.052872 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:12.052883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.054102 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.054109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:12.054317 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.054356 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:49:12.054460 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:49:12.054611 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":9,"thread_start_us":573,"threads_started":1}
I20260619 02:49:12.055153 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.048195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:37837 (local address 127.7.246.66:34511)
0619 02:49:12.049213 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:12.049244 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:12.049375 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:12.049837 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:12.049843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:12.049882 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:12.050383 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:12.050395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.051477 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.051480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:12.052877 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:12.052888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.054165 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.054172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:12.054314 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.054360 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:49:12.054460 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:49:12.054615 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":67,"thread_start_us":662,"threads_started":1}
I20260619 02:49:12.056285 26807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bb1a546414339ad1201bc5ecba0dc" candidate_uuid: "27a01d61ace24654bd59d33e06a1b747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b667acaf5842b590581170fd1f0808" is_pre_election: true
I20260619 02:49:12.056268 26540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bb1a546414339ad1201bc5ecba0dc" candidate_uuid: "27a01d61ace24654bd59d33e06a1b747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc1060952314479af180c184836cc24" is_pre_election: true
I20260619 02:49:12.056803 26807 raft_consensus.cc:2454] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a01d61ace24654bd59d33e06a1b747 in term 0.
I20260619 02:49:12.056803 26540 raft_consensus.cc:2454] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a01d61ace24654bd59d33e06a1b747 in term 0.
I20260619 02:49:12.057502 26608 leader_election.cc:302] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a01d61ace24654bd59d33e06a1b747, 2cc1060952314479af180c184836cc24; no voters:
I20260619 02:49:12.057955 26876 raft_consensus.cc:2781] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:12.058109 26876 raft_consensus.cc:482] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:12.058215 26876 raft_consensus.cc:3037] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:12.060411 26876 raft_consensus.cc:504] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:12.060958 26876 leader_election.cc:288] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [CANDIDATE]: Term 1 election: Requested vote from peers 19b667acaf5842b590581170fd1f0808 (127.7.246.67:40141), 2cc1060952314479af180c184836cc24 (127.7.246.65:37837)
I20260619 02:49:12.061452 26807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bb1a546414339ad1201bc5ecba0dc" candidate_uuid: "27a01d61ace24654bd59d33e06a1b747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b667acaf5842b590581170fd1f0808"
I20260619 02:49:12.061463 26540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2bb1a546414339ad1201bc5ecba0dc" candidate_uuid: "27a01d61ace24654bd59d33e06a1b747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc1060952314479af180c184836cc24"
I20260619 02:49:12.061682 26540 raft_consensus.cc:3037] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:12.061751 26807 raft_consensus.cc:3037] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:12.063862 26540 raft_consensus.cc:2454] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a01d61ace24654bd59d33e06a1b747 in term 1.
I20260619 02:49:12.063933 26807 raft_consensus.cc:2454] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a01d61ace24654bd59d33e06a1b747 in term 1.
I20260619 02:49:12.064249 26608 leader_election.cc:302] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a01d61ace24654bd59d33e06a1b747, 2cc1060952314479af180c184836cc24; no voters:
I20260619 02:49:12.064591 26876 raft_consensus.cc:2781] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:12.065003 26876 raft_consensus.cc:686] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 1 LEADER]: Becoming Leader. State: Replica: 27a01d61ace24654bd59d33e06a1b747, State: Running, Role: LEADER
I20260619 02:49:12.065460 26876 consensus_queue.cc:237] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } }
I20260619 02:49:12.070343 26396 catalog_manager.cc:5495] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a01d61ace24654bd59d33e06a1b747 (127.7.246.66). New cstate: current_term: 1 leader_uuid: "27a01d61ace24654bd59d33e06a1b747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a01d61ace24654bd59d33e06a1b747" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 43439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:12.192015 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.186483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.7.246.66:43439)
0619 02:49:12.186931 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.186935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.186953 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:49:12.187178 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.187182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.187282 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.187422 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:12.187918 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.188605 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.190022 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.190397 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.190497 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.191022 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:12.191505 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:12.191630 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.191751 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.191789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
I20260619 02:49:12.199497 26540 raft_consensus.cc:1261] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 1 FOLLOWER]: Refusing update from remote peer 27a01d61ace24654bd59d33e06a1b747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:12.199558 26807 raft_consensus.cc:1261] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 1 FOLLOWER]: Refusing update from remote peer 27a01d61ace24654bd59d33e06a1b747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:12.200402 26881 consensus_queue.cc:1035] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc1060952314479af180c184836cc24" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:12.200736 26876 consensus_queue.cc:1035] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b667acaf5842b590581170fd1f0808" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:12.214494 26889 mvcc.cc:205] Tried to move back new op lower bound from 7298405794599624704 to 7298405794064355328. Current Snapshot: MvccSnapshot[applied={T|T < 7298405794599624704 or (T in {7298405794599624704})}]
I20260619 02:49:12.215161 26890 mvcc.cc:205] Tried to move back new op lower bound from 7298405794599624704 to 7298405794064355328. Current Snapshot: MvccSnapshot[applied={T|T < 7298405794599624704}]
I20260619 02:49:12.221998 26396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:12.222273 26396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:12.225806 26396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0605fca01464b0899e15b159e5f8760: Sending DeleteTablet for 3 replicas of tablet 9b2bb1a546414339ad1201bc5ecba0dc
I20260619 02:49:12.226912 26787 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:12 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47266
I20260619 02:49:12.226975 26654 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:12 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:58114
I20260619 02:49:12.227162 26520 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2bb1a546414339ad1201bc5ecba0dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:12 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:47144
I20260619 02:49:12.228965 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26457
I20260619 02:49:12.239084 26898 tablet_replica.cc:330] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: stopping tablet replica
I20260619 02:49:12.239272 26899 tablet_replica.cc:330] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: stopping tablet replica
I20260619 02:49:12.239528 26898 raft_consensus.cc:2229] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:12.239746 26899 raft_consensus.cc:2229] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:12.240206 26898 raft_consensus.cc:2258] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:12.240267 26899 raft_consensus.cc:2258] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:12.242141 26899 ts_tablet_manager.cc:1905] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:12.242146 26898 ts_tablet_manager.cc:1905] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:12.249593 26899 ts_tablet_manager.cc:1918] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:12.249599 26898 ts_tablet_manager.cc:1918] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:12.249753 26898 log.cc:1196] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/9b2bb1a546414339ad1201bc5ecba0dc
I20260619 02:49:12.249786 26899 log.cc:1196] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/9b2bb1a546414339ad1201bc5ecba0dc
I20260619 02:49:12.250372 26898 ts_tablet_manager.cc:1939] T 9b2bb1a546414339ad1201bc5ecba0dc P 27a01d61ace24654bd59d33e06a1b747: Deleting consensus metadata
I20260619 02:49:12.250772 26899 ts_tablet_manager.cc:1939] T 9b2bb1a546414339ad1201bc5ecba0dc P 2cc1060952314479af180c184836cc24: Deleting consensus metadata
I20260619 02:49:12.252271 26380 catalog_manager.cc:4841] TS 27a01d61ace24654bd59d33e06a1b747 (127.7.246.66:43439): tablet 9b2bb1a546414339ad1201bc5ecba0dc (table test-table [id=ba214682924749c381e0bc92d337a61a]) successfully deleted
I20260619 02:49:12.252529 26381 catalog_manager.cc:4841] TS 2cc1060952314479af180c184836cc24 (127.7.246.65:37837): tablet 9b2bb1a546414339ad1201bc5ecba0dc (table test-table [id=ba214682924749c381e0bc92d337a61a]) successfully deleted
I20260619 02:49:12.253923 26897 tablet_replica.cc:330] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: stopping tablet replica
I20260619 02:49:12.254302 26897 raft_consensus.cc:2229] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:12.254760 26897 raft_consensus.cc:2258] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:12.256242 26897 ts_tablet_manager.cc:1905] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:12.261324 26897 ts_tablet_manager.cc:1918] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:12.261493 26897 log.cc:1196] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/9b2bb1a546414339ad1201bc5ecba0dc
I20260619 02:49:12.262064 26897 ts_tablet_manager.cc:1939] T 9b2bb1a546414339ad1201bc5ecba0dc P 19b667acaf5842b590581170fd1f0808: Deleting consensus metadata
Jun 19 02:49:12 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:49:12.263619 26380 catalog_manager.cc:4841] TS 19b667acaf5842b590581170fd1f0808 (127.7.246.67:40141): tablet 9b2bb1a546414339ad1201bc5ecba0dc (table test-table [id=ba214682924749c381e0bc92d337a61a]) successfully deleted
I20260619 02:49:12.270040 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.230187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.7.246.65:37837)
0619 02:49:12.230372 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.230379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.253221 (+ 22842us) server_negotiation.cc:407] Connection header received
0619 02:49:12.253440 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.253448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.253540 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.253656 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:12.255368 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.256493 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.258536 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.259180 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.259328 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.264361 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:12.264372 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:12.264385 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:12.264420 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:12.267250 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.267896 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.267903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.267905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.268026 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.268422 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.268431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.268433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.268767 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:12.269135 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.269402 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.269664 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:49:12.270802 26580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:12.378837 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26457
I20260619 02:49:12.396610 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26591
Jun 19 02:49:12 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:49:12.413735 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.397404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.7.246.66:43439)
0619 02:49:12.397611 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.397617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.397639 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:49:12.397854 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.397860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.397944 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.398053 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:12.399309 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.400493 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.401905 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.402601 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.402757 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.406791 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:12.406808 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:12.406828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:12.406874 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:12.410878 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.411535 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.411544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.411549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.411753 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.412092 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.412097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.412101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.412550 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:12.412885 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.413069 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.413321 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:49:12.414438 26714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:12.527500 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26591
I20260619 02:49:12.546893 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26724
Jun 19 02:49:12 dist-test-slave-9tkz krb5kdc[26351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:49:12.564069 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:12.547737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.7.246.67:40141)
0619 02:49:12.548253 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:12.548257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:12.548274 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:12.548325 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:12.548329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:12.548390 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:12.548508 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:12.550232 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.551026 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.552417 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:12.552879 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:12.553022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:12.558000 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:12.558012 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:12.558025 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:12.558065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:12.561105 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.561895 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.561904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.561906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.562039 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:12.562458 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:12.562463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:12.562465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:12.562855 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:12.563177 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:12.563488 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 02:49:12.563714 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260619 02:49:12.564746 26847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:12.658512 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26724
I20260619 02:49:12.675379 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26364
I20260619 02:49:12.676569 26426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:12.787007 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26364
2026-06-19T02:49:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4642 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T02:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:12Z Disabled control of system clock
I20260619 02:49:12.862797 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:44543
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:44543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:13.183265 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:13.183542 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:13.183622 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:13.190663 26908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:13.190768 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:13.190804 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:13.190862 26908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:13.190907 26908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:13.199868 26908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:44543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:44543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:13.201073 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:13.202754 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:13.214294 26913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.214367 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.214329 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:13.215467 26908 server_base.cc:1034] running on GCE node
I20260619 02:49:13.216152 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:13.217459 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:13.218663 26908 hybrid_clock.cc:648] HybridClock initialized: now 1781837353218618 us; error 37 us; skew 500 ppm
I20260619 02:49:13.218968 26908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:13.221118 26908 webserver.cc:460] Webserver started at http://127.7.246.126:36267/ using document root <none> and password file <none>
I20260619 02:49:13.221803 26908 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:13.221896 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:13.222157 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:13.224936 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "3d92695c14874a82a899cee770d933c4"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "9c3323cf34b38f41412fdf9b3521f2d3"
server_key_iv: "8585477cd3b348a6c10dcfe2114982fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.225732 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "3d92695c14874a82a899cee770d933c4"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "9c3323cf34b38f41412fdf9b3521f2d3"
server_key_iv: "8585477cd3b348a6c10dcfe2114982fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.230533 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:49:13.233803 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:13.235199 26908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:49:13.235337 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "3d92695c14874a82a899cee770d933c4"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "9c3323cf34b38f41412fdf9b3521f2d3"
server_key_iv: "8585477cd3b348a6c10dcfe2114982fe"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.235508 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:13.250998 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:13.251739 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:13.251971 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:13.270476 26908 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:44543
I20260619 02:49:13.270496 26973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:44543 every 8 connection(s)
I20260619 02:49:13.271827 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:13.274405 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26908
I20260619 02:49:13.274587 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:13.275170 8153 external_mini_cluster.cc:1428] Setting key b61909e51e99a56b6b05f5b11f0bd8f9
I20260619 02:49:13.276185 26974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:13.287825 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:13.276896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.7.246.126:44543)
0619 02:49:13.277799 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:13.277810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:13.277850 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:13.279441 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:13.279462 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:13.279904 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:13.280312 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:13.280844 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.282230 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.283543 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.284169 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.284368 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:13.285527 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:13.285534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:13.285595 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:13.286198 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:13.286268 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:13.286293 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:13.286422 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:13.286648 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 02:49:13.286942 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260619 02:49:13.287993 26974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: Bootstrap starting.
I20260619 02:49:13.291604 26974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:13.292812 26974 log.cc:824] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:13.295423 26974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: No bootstrap required, opened a new log
I20260619 02:49:13.302873 26974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } }
I20260619 02:49:13.303175 26974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:13.303258 26974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d92695c14874a82a899cee770d933c4, State: Initialized, Role: FOLLOWER
I20260619 02:49:13.303845 26974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } }
I20260619 02:49:13.304093 26974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:13.304236 26974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:13.304397 26974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:13.306505 26974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } }
I20260619 02:49:13.306922 26974 leader_election.cc:302] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d92695c14874a82a899cee770d933c4; no voters:
I20260619 02:49:13.307439 26974 leader_election.cc:288] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:13.307575 26979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:13.308290 26979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d92695c14874a82a899cee770d933c4, State: Running, Role: LEADER
I20260619 02:49:13.308800 26979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } }
I20260619 02:49:13.309624 26974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:13.311568 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d92695c14874a82a899cee770d933c4. Latest consensus state: current_term: 1 leader_uuid: "3d92695c14874a82a899cee770d933c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } } }
I20260619 02:49:13.311955 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:13.314523 26980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d92695c14874a82a899cee770d933c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d92695c14874a82a899cee770d933c4" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 44543 } } }
I20260619 02:49:13.314769 26980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:13.314790 26986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:13.321573 26986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:13.330250 26986 catalog_manager.cc:1347] Generated new cluster ID: 1dc7c5fc118f423a863b93b674fb7688
I20260619 02:49:13.330374 26986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:13.350409 26986 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:13.351691 26986 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:13.360554 26986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: Generated new TSK 0
I20260619 02:49:13.361392 26986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:13.366026 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--builtin_ntp_servers=127.7.246.84:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:13.749250 26998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:13.749593 26998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:13.749794 26998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:13.757051 26998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:13.757181 26998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:13.757357 26998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:13.766640 26998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:13.767931 26998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:13.769764 26998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:13.783363 27006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.783324 27004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:13.783412 27003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:13.784801 26998 server_base.cc:1034] running on GCE node
I20260619 02:49:13.785356 26998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:13.786381 26998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:13.787585 26998 hybrid_clock.cc:648] HybridClock initialized: now 1781837353787518 us; error 61 us; skew 500 ppm
I20260619 02:49:13.787914 26998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:13.790262 26998 webserver.cc:460] Webserver started at http://127.7.246.65:33821/ using document root <none> and password file <none>
I20260619 02:49:13.790966 26998 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:13.791079 26998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:13.791422 26998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:13.794443 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "3499088de1f44c70b53ea42a8bbfc708"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "93b8ae3af5f0ba5c6f42eb21cd3a6fb7"
server_key_iv: "bccfe2fd61748da8dbdec8a9a95de3b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.795221 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "3499088de1f44c70b53ea42a8bbfc708"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "93b8ae3af5f0ba5c6f42eb21cd3a6fb7"
server_key_iv: "bccfe2fd61748da8dbdec8a9a95de3b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.800184 26998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:13.803536 27012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:13.804836 26998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 02:49:13.805008 26998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "3499088de1f44c70b53ea42a8bbfc708"
format_stamp: "Formatted at 2026-06-19 02:49:13 on dist-test-slave-9tkz"
server_key: "93b8ae3af5f0ba5c6f42eb21cd3a6fb7"
server_key_iv: "bccfe2fd61748da8dbdec8a9a95de3b6"
server_key_version: "encryptionkey@0"
I20260619 02:49:13.805220 26998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:13.818816 26998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:13.819622 26998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:13.819934 26998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:13.820840 26998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:13.822434 26998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:13.822549 26998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:13.822682 26998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:13.822774 26998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:13.855733 26998 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:43761
I20260619 02:49:13.855774 27124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:43761 every 8 connection(s)
I20260619 02:49:13.857035 26998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:13.864341 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 26998
I20260619 02:49:13.864508 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:13.864949 8153 external_mini_cluster.cc:1428] Setting key b9928410dfda90764568c10be710459d
I20260619 02:49:13.867376 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--builtin_ntp_servers=127.7.246.84:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:13.869985 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:13.859609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:35045 (local address 127.7.246.126:44543)
0619 02:49:13.860143 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:13.860151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:13.861395 (+ 1244us) server_negotiation.cc:407] Connection header received
0619 02:49:13.863377 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:13.863383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:13.863475 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:13.863597 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:13.864766 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.865530 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.867098 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.867585 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.867747 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:13.869107 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:13.869111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:13.869115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:13.869131 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:13.869170 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:13.869184 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:13.869281 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:13.869493 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 02:49:13.869608 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260619 02:49:13.870980 27128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:13.860254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:44543 (local address 127.7.246.65:35045)
0619 02:49:13.861234 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:13.861272 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:13.863033 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:13.863851 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:13.863864 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:13.863952 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:13.864596 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:13.864627 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.865726 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.865732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:13.866953 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:13.866965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:13.867826 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:13.867833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:13.868062 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:13.868795 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:13.868798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:13.868836 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:13.868908 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:13.869279 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:13.869291 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:13.869301 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:13.869483 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 02:49:13.869939 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":286,"threads_started":1}
I20260619 02:49:13.872541 27125 heartbeater.cc:347] Connected to a master server at 127.7.246.126:44543
I20260619 02:49:13.872871 27125 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:13.873639 27125 heartbeater.cc:511] Master 127.7.246.126:44543 requested a full tablet report, sending...
I20260619 02:49:13.875787 26939 ts_manager.cc:194] Registered new tserver with Master: 3499088de1f44c70b53ea42a8bbfc708 (127.7.246.65:43761)
I20260619 02:49:13.877951 26939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.65:35045
W20260619 02:49:14.184274 27129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:14.184561 27129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:14.184648 27129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:14.192121 27129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:14.192224 27129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:14.192328 27129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:14.201591 27129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:14.202786 27129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:14.204511 27129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:14.217820 27135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.217890 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.217912 27137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.219317 27129 server_base.cc:1034] running on GCE node
I20260619 02:49:14.219857 27129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:14.220741 27129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:14.221918 27129 hybrid_clock.cc:648] HybridClock initialized: now 1781837354221857 us; error 51 us; skew 500 ppm
I20260619 02:49:14.222236 27129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:14.224304 27129 webserver.cc:460] Webserver started at http://127.7.246.66:46801/ using document root <none> and password file <none>
I20260619 02:49:14.224967 27129 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:14.225083 27129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:14.225414 27129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:14.228195 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "5623b86cd473438db6cf7172109c4a95"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "d1147abab715ef87e2f80037edd7cce0"
server_key_iv: "aca74c9bd628cf8922e38a121f395d06"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.228977 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "5623b86cd473438db6cf7172109c4a95"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "d1147abab715ef87e2f80037edd7cce0"
server_key_iv: "aca74c9bd628cf8922e38a121f395d06"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.233857 27129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 02:49:14.237262 27143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.238914 27129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:14.239136 27129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "5623b86cd473438db6cf7172109c4a95"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "d1147abab715ef87e2f80037edd7cce0"
server_key_iv: "aca74c9bd628cf8922e38a121f395d06"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.239434 27129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:14.261088 27129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:14.262075 27129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:14.262444 27129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:14.263494 27129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:14.265153 27129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:14.265295 27129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.265451 27129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:14.265578 27129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.307061 27129 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:44451
I20260619 02:49:14.307127 27255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:44451 every 8 connection(s)
I20260619 02:49:14.308683 27129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:14.312533 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27129
I20260619 02:49:14.312706 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:14.313117 8153 external_mini_cluster.cc:1428] Setting key fb3e50909d3fc5adc8d22a1dc7fde6ca
I20260619 02:49:14.315641 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--builtin_ntp_servers=127.7.246.84:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:14.322948 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.311673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:56257 (local address 127.7.246.126:44543)
0619 02:49:14.311867 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.311872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.313601 (+ 1729us) server_negotiation.cc:407] Connection header received
0619 02:49:14.315661 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.315668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.315767 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.315882 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.317045 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.318061 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.319982 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.320485 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.320656 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.322224 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.322228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.322231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:14.322248 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.322277 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:14.322289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.322411 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.322647 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.322719 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:49:14.324054 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.312195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:44543 (local address 127.7.246.66:56257)
0619 02:49:14.313418 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.313475 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.315368 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.316131 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.316141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.316234 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:14.316856 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.316874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.318246 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.318254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.319806 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.319820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.320758 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.320768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.321027 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.321851 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:14.321854 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:14.321890 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:14.322024 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:14.322419 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:14.322432 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:14.322441 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.322631 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.323066 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":460,"threads_started":1}
I20260619 02:49:14.325783 27256 heartbeater.cc:347] Connected to a master server at 127.7.246.126:44543
I20260619 02:49:14.326094 27256 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:14.326912 27256 heartbeater.cc:511] Master 127.7.246.126:44543 requested a full tablet report, sending...
I20260619 02:49:14.328450 26939 ts_manager.cc:194] Registered new tserver with Master: 5623b86cd473438db6cf7172109c4a95 (127.7.246.66:44451)
I20260619 02:49:14.329713 26939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.66:56257
W20260619 02:49:14.636445 27259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:14.636701 27259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:14.636796 27259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:14.643987 27259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:14.644071 27259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:14.644169 27259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:14.653033 27259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:44543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:14.654153 27259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:14.655756 27259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:14.668619 27264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.668641 27267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:14.668581 27265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:14.670208 27259 server_base.cc:1034] running on GCE node
I20260619 02:49:14.670871 27259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:14.672040 27259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:14.673287 27259 hybrid_clock.cc:648] HybridClock initialized: now 1781837354673241 us; error 48 us; skew 500 ppm
I20260619 02:49:14.673689 27259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:14.675882 27259 webserver.cc:460] Webserver started at http://127.7.246.67:35953/ using document root <none> and password file <none>
I20260619 02:49:14.676550 27259 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:14.676666 27259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:14.676995 27259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:14.679930 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "911fd5a66c72d17f0774249fb5127a51"
server_key_iv: "32ed1a6b8e801e53aa01678837722d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.680683 27259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "911fd5a66c72d17f0774249fb5127a51"
server_key_iv: "32ed1a6b8e801e53aa01678837722d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.685479 27259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 02:49:14.688838 27273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.690114 27259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:14.690284 27259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb"
format_stamp: "Formatted at 2026-06-19 02:49:14 on dist-test-slave-9tkz"
server_key: "911fd5a66c72d17f0774249fb5127a51"
server_key_iv: "32ed1a6b8e801e53aa01678837722d17"
server_key_version: "encryptionkey@0"
I20260619 02:49:14.690488 27259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:14.708514 27259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:14.709326 27259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:14.709717 27259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:14.710678 27259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:14.712036 27259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:14.712163 27259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.712288 27259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:14.712384 27259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:14.745250 27259 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:39597
I20260619 02:49:14.745304 27385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:39597 every 8 connection(s)
I20260619 02:49:14.746731 27259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:14.750782 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27259
I20260619 02:49:14.750962 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:14.751521 8153 external_mini_cluster.cc:1428] Setting key bb35ff8c4658fb552d5e0eb59f38507b
I20260619 02:49:14.759461 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.749260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:58539 (local address 127.7.246.126:44543)
0619 02:49:14.749453 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.749458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.751071 (+ 1613us) server_negotiation.cc:407] Connection header received
0619 02:49:14.753480 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.753487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.753595 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.753701 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.754838 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.755533 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.757088 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.757462 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.757601 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.758780 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.758785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.758788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:14.758802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.758831 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:14.758842 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.758908 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.759123 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.759216 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:49:14.760360 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.749930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:44543 (local address 127.7.246.67:58539)
0619 02:49:14.750897 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.750957 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.753190 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.753888 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.753896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.754008 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:14.754690 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.754707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.755648 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.755651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.756948 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.756960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.757673 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.757678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.757857 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.758520 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:14.758522 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:14.758557 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:14.758629 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:14.758922 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:14.758929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:14.758934 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.759094 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.759391 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260619 02:49:14.761641 27386 heartbeater.cc:347] Connected to a master server at 127.7.246.126:44543
I20260619 02:49:14.761936 27386 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:14.762614 27386 heartbeater.cc:511] Master 127.7.246.126:44543 requested a full tablet report, sending...
I20260619 02:49:14.763895 26939 ts_manager.cc:194] Registered new tserver with Master: bb3f7bf9227c4d1e9934caa4ae71d6eb (127.7.246.67:39597)
I20260619 02:49:14.764740 26939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.67:58539
I20260619 02:49:14.766568 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:14.776234 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.770764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.7.246.126:44543)
0619 02:49:14.770911 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.770915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.771358 (+ 443us) server_negotiation.cc:407] Connection header received
0619 02:49:14.771468 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.771472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.771526 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.771596 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:14.772029 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.772656 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.774009 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.774396 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.774493 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.775525 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:14.775529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:14.775531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:14.775546 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:14.775572 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:14.775583 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:14.775684 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.775864 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.775934 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:49:14.779881 26939 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:14.786477 26939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:14.808768 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.798169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44451 (local address 127.0.0.1:51542)
0619 02:49:14.800079 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.800091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.800161 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.802448 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.802456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.802493 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:14.802901 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.802912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.804429 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.804437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.805994 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.806009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.807821 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.807828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.808023 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.808090 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.808211 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.808385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"mutex_wait_us":1,"thread_start_us":1725,"threads_started":1}
I20260619 02:49:14.808758 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.798527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.7.246.66:44451)
0619 02:49:14.801879 (+ 3352us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.801887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.801912 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:49:14.802013 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.802021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.802237 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.802432 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:14.803036 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.804295 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.806189 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.807620 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.807763 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.808033 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.808276 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.808427 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3172,"thread_start_us":3058,"threads_started":1}
I20260619 02:49:14.809356 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.797624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:39597 (local address 127.0.0.1:43858)
0619 02:49:14.799624 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.799637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.799750 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.801222 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.801229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.801283 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:14.801675 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.801689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.803961 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.803968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.806415 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.806432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.808512 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.808518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.808703 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.808741 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.808821 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.808996 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1875,"mutex_wait_us":35,"thread_start_us":1744,"threads_started":1}
I20260619 02:49:14.810375 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.798009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.7.246.67:39597)
0619 02:49:14.800677 (+ 2668us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.800685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.800718 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:14.800822 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.800831 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.801037 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.801218 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:14.801892 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.803795 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.806655 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.808005 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.808127 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.808392 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.809768 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.810017 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2477,"thread_start_us":2355,"threads_started":1}
I20260619 02:49:14.810576 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.797407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43761 (local address 127.0.0.1:39172)
0619 02:49:14.798950 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.798989 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.799153 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.803530 (+ 4377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.803535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.803558 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:14.803927 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.803940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.806027 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.806035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.807953 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.807966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.809787 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.809796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.809983 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.810012 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.810097 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.810208 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":3,"thread_start_us":1160,"threads_started":1}
I20260619 02:49:14.810639 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.797626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.7.246.65:43761)
0619 02:49:14.803008 (+ 5382us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.803016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.803043 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:49:14.803131 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.803139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.803340 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.804143 (+ 803us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:14.804354 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.805884 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.808117 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.809538 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.809737 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.809979 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.810144 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.810294 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5228,"thread_start_us":5110,"threads_started":1}
I20260619 02:49:14.814186 27060 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb19779d254865ba499708fe266bce (DEFAULT_TABLE table=test-table [id=2723c280b0c04032905676c6c659bcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:14.814100 27191 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb19779d254865ba499708fe266bce (DEFAULT_TABLE table=test-table [id=2723c280b0c04032905676c6c659bcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:14.814100 27321 tablet_service.cc:1467] Processing CreateTablet for tablet 2deb19779d254865ba499708fe266bce (DEFAULT_TABLE table=test-table [id=2723c280b0c04032905676c6c659bcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:14.815362 27060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb19779d254865ba499708fe266bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:14.815472 27191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb19779d254865ba499708fe266bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:14.815471 27321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2deb19779d254865ba499708fe266bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:14.826498 27404 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Bootstrap starting.
I20260619 02:49:14.829391 27405 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Bootstrap starting.
I20260619 02:49:14.830229 27404 tablet_bootstrap.cc:654] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:14.831575 27404 log.cc:824] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:14.832728 27405 tablet_bootstrap.cc:654] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:14.833026 27406 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Bootstrap starting.
I20260619 02:49:14.833951 27405 log.cc:824] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:14.834877 27404 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: No bootstrap required, opened a new log
I20260619 02:49:14.835137 27404 ts_tablet_manager.cc:1397] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260619 02:49:14.836500 27406 tablet_bootstrap.cc:654] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:14.836879 27405 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: No bootstrap required, opened a new log
I20260619 02:49:14.837123 27405 ts_tablet_manager.cc:1397] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 02:49:14.837509 27406 log.cc:824] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:14.840332 27406 tablet_bootstrap.cc:492] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: No bootstrap required, opened a new log
I20260619 02:49:14.840576 27406 ts_tablet_manager.cc:1397] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 02:49:14.845516 27404 raft_consensus.cc:348] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.846004 27404 raft_consensus.cc:374] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:14.846092 27404 raft_consensus.cc:729] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3499088de1f44c70b53ea42a8bbfc708, State: Initialized, Role: FOLLOWER
I20260619 02:49:14.846673 27404 consensus_queue.cc:260] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.847728 27125 heartbeater.cc:503] Master 127.7.246.126:44543 was elected leader, sending a full tablet report...
I20260619 02:49:14.847764 27406 raft_consensus.cc:348] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.848124 27406 raft_consensus.cc:374] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:14.848232 27406 raft_consensus.cc:729] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3f7bf9227c4d1e9934caa4ae71d6eb, State: Initialized, Role: FOLLOWER
I20260619 02:49:14.848369 27404 ts_tablet_manager.cc:1428] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 02:49:14.847445 27405 raft_consensus.cc:348] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.848766 27406 consensus_queue.cc:260] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.848922 27405 raft_consensus.cc:374] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:14.849015 27405 raft_consensus.cc:729] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5623b86cd473438db6cf7172109c4a95, State: Initialized, Role: FOLLOWER
I20260619 02:49:14.849627 27405 consensus_queue.cc:260] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.854631 27256 heartbeater.cc:503] Master 127.7.246.126:44543 was elected leader, sending a full tablet report...
I20260619 02:49:14.854846 27405 ts_tablet_manager.cc:1428] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260619 02:49:14.855028 27386 heartbeater.cc:503] Master 127.7.246.126:44543 was elected leader, sending a full tablet report...
I20260619 02:49:14.856240 27406 ts_tablet_manager.cc:1428] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Time spent starting tablet: real 0.016s user 0.003s sys 0.013s
W20260619 02:49:14.859318 27126 tablet.cc:2369] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:14.895200 27411 raft_consensus.cc:482] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:14.895541 27411 raft_consensus.cc:504] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.897320 27411 leader_election.cc:288] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3499088de1f44c70b53ea42a8bbfc708 (127.7.246.65:43761), 5623b86cd473438db6cf7172109c4a95 (127.7.246.66:44451)
I20260619 02:49:14.903661 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.897758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:45981 (local address 127.7.246.65:43761)
0619 02:49:14.897981 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.897989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.898015 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:49:14.898211 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.898215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.898296 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.898410 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:14.898888 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.899825 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.902030 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.902757 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.902929 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.903001 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.903267 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.903400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:49:14.903699 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.897509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:43761 (local address 127.7.246.67:45981)
0619 02:49:14.897874 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.897898 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.898028 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.898458 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.898462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.898489 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:14.898715 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.898726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.899971 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.899976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.901864 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.901876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.902911 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.902919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.903096 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.903119 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.903224 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.903369 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1}
I20260619 02:49:14.903965 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.897826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:46239 (local address 127.7.246.66:44451)
0619 02:49:14.898006 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.898012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.898631 (+ 619us) server_negotiation.cc:407] Connection header received
0619 02:49:14.898748 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.898753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.898838 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.898924 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:14.899505 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.900406 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.902387 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.903112 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.903290 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.903368 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.903624 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.903740 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:49:14.904049 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.897787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:44451 (local address 127.7.246.67:46239)
0619 02:49:14.898521 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:14.898541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:14.898646 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:14.898962 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:14.898969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:14.898995 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:14.899362 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.899374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.900594 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.900601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.902198 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:14.902210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.903279 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.903287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:14.903460 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.903482 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:49:14.903584 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:49:14.903705 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":457,"threads_started":1}
I20260619 02:49:14.904675 27080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb19779d254865ba499708fe266bce" candidate_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3499088de1f44c70b53ea42a8bbfc708" is_pre_election: true
I20260619 02:49:14.904886 27211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb19779d254865ba499708fe266bce" candidate_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5623b86cd473438db6cf7172109c4a95" is_pre_election: true
I20260619 02:49:14.905218 27080 raft_consensus.cc:2454] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb3f7bf9227c4d1e9934caa4ae71d6eb in term 0.
I20260619 02:49:14.905397 27211 raft_consensus.cc:2454] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb3f7bf9227c4d1e9934caa4ae71d6eb in term 0.
I20260619 02:49:14.905915 27276 leader_election.cc:302] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3499088de1f44c70b53ea42a8bbfc708, bb3f7bf9227c4d1e9934caa4ae71d6eb; no voters:
I20260619 02:49:14.906267 27411 raft_consensus.cc:2781] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:14.906388 27411 raft_consensus.cc:482] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:14.906509 27411 raft_consensus.cc:3037] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:14.908806 27411 raft_consensus.cc:504] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.909518 27411 leader_election.cc:288] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [CANDIDATE]: Term 1 election: Requested vote from peers 3499088de1f44c70b53ea42a8bbfc708 (127.7.246.65:43761), 5623b86cd473438db6cf7172109c4a95 (127.7.246.66:44451)
I20260619 02:49:14.909930 27080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb19779d254865ba499708fe266bce" candidate_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3499088de1f44c70b53ea42a8bbfc708"
I20260619 02:49:14.909977 27211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2deb19779d254865ba499708fe266bce" candidate_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5623b86cd473438db6cf7172109c4a95"
I20260619 02:49:14.910113 27080 raft_consensus.cc:3037] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:14.910190 27211 raft_consensus.cc:3037] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:14.912338 27080 raft_consensus.cc:2454] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb3f7bf9227c4d1e9934caa4ae71d6eb in term 1.
I20260619 02:49:14.912453 27211 raft_consensus.cc:2454] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb3f7bf9227c4d1e9934caa4ae71d6eb in term 1.
I20260619 02:49:14.912720 27276 leader_election.cc:302] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3499088de1f44c70b53ea42a8bbfc708, bb3f7bf9227c4d1e9934caa4ae71d6eb; no voters:
I20260619 02:49:14.913009 27411 raft_consensus.cc:2781] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:14.913439 27411 raft_consensus.cc:686] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 1 LEADER]: Becoming Leader. State: Replica: bb3f7bf9227c4d1e9934caa4ae71d6eb, State: Running, Role: LEADER
I20260619 02:49:14.914004 27411 consensus_queue.cc:237] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } }
I20260619 02:49:14.918429 26939 catalog_manager.cc:5495] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb reported cstate change: term changed from 0 to 1, leader changed from <none> to bb3f7bf9227c4d1e9934caa4ae71d6eb (127.7.246.67). New cstate: current_term: 1 leader_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb3f7bf9227c4d1e9934caa4ae71d6eb" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 39597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:14.937731 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.931690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.7.246.67:39597)
0619 02:49:14.931923 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.931929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.932043 (+ 114us) server_negotiation.cc:407] Connection header received
0619 02:49:14.932311 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:14.932315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:14.932389 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:14.932544 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:14.932996 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.933796 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.935303 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:14.935859 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:14.936003 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:14.936588 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:14.937113 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:14.937314 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:14.937425 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 02:49:14.937468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:49:14.945888 27080 raft_consensus.cc:1261] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 1 FOLLOWER]: Refusing update from remote peer bb3f7bf9227c4d1e9934caa4ae71d6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:14.945878 27211 raft_consensus.cc:1261] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 1 FOLLOWER]: Refusing update from remote peer bb3f7bf9227c4d1e9934caa4ae71d6eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:14.946848 27414 consensus_queue.cc:1035] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5623b86cd473438db6cf7172109c4a95" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:14.947173 27411 consensus_queue.cc:1035] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3499088de1f44c70b53ea42a8bbfc708" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 43761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:14.960069 27419 mvcc.cc:205] Tried to move back new op lower bound from 7298405805847920640 to 7298405805731930112. Current Snapshot: MvccSnapshot[applied={T|T < 7298405805847920640}]
I20260619 02:49:14.969892 26939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:14.970160 26939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:14.973623 26939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d92695c14874a82a899cee770d933c4: Sending DeleteTablet for 3 replicas of tablet 2deb19779d254865ba499708fe266bce
I20260619 02:49:14.974754 27060 tablet_service.cc:1514] Processing DeleteTablet for tablet 2deb19779d254865ba499708fe266bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:14 UTC) from {username='slave'} at 127.0.0.1:39172
I20260619 02:49:14.974800 27321 tablet_service.cc:1514] Processing DeleteTablet for tablet 2deb19779d254865ba499708fe266bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:14 UTC) from {username='slave'} at 127.0.0.1:43858
I20260619 02:49:14.975477 27191 tablet_service.cc:1514] Processing DeleteTablet for tablet 2deb19779d254865ba499708fe266bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:14 UTC) from {username='slave'} at 127.0.0.1:51542
I20260619 02:49:14.976660 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26998
I20260619 02:49:15.000555 27428 tablet_replica.cc:330] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: stopping tablet replica
I20260619 02:49:15.001623 27428 raft_consensus.cc:2229] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:15.002414 27428 raft_consensus.cc:2258] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:15.003409 27426 tablet_replica.cc:330] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: stopping tablet replica
I20260619 02:49:15.003973 27426 raft_consensus.cc:2229] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:15.004769 27428 ts_tablet_manager.cc:1905] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:15.004779 27426 raft_consensus.cc:2258] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:15.005765 27427 tablet_replica.cc:330] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: stopping tablet replica
I20260619 02:49:15.006142 27427 raft_consensus.cc:2229] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:15.006618 27427 raft_consensus.cc:2258] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:15.007615 27426 ts_tablet_manager.cc:1905] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:15.008682 27427 ts_tablet_manager.cc:1905] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:15.014213 27428 ts_tablet_manager.cc:1918] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:15.014369 27428 log.cc:1196] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/2deb19779d254865ba499708fe266bce
I20260619 02:49:15.014869 27428 ts_tablet_manager.cc:1939] T 2deb19779d254865ba499708fe266bce P 5623b86cd473438db6cf7172109c4a95: Deleting consensus metadata
I20260619 02:49:15.015570 27426 ts_tablet_manager.cc:1918] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:15.015800 27426 log.cc:1196] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/2deb19779d254865ba499708fe266bce
I20260619 02:49:15.016410 27426 ts_tablet_manager.cc:1939] T 2deb19779d254865ba499708fe266bce P bb3f7bf9227c4d1e9934caa4ae71d6eb: Deleting consensus metadata
I20260619 02:49:15.016942 26926 catalog_manager.cc:4841] TS 5623b86cd473438db6cf7172109c4a95 (127.7.246.66:44451): tablet 2deb19779d254865ba499708fe266bce (table test-table [id=2723c280b0c04032905676c6c659bcfa]) successfully deleted
I20260619 02:49:15.018042 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:14.978577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.7.246.65:43761)
0619 02:49:14.978842 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:14.978851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:14.999407 (+ 20556us) server_negotiation.cc:407] Connection header received
0619 02:49:15.001062 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:15.001069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:15.001186 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:15.001301 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:15.001985 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.003196 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.005048 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.005810 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.005924 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:15.011409 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:15.011419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:15.011434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:15.012154 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:15.012212 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:15.012237 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:15.012325 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:15.017501 (+ 5176us) server_negotiation.cc:299] Negotiation successful
0619 02:49:15.017660 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:49:15.018558 27427 ts_tablet_manager.cc:1918] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:15.018731 27427 log.cc:1196] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/2deb19779d254865ba499708fe266bce
I20260619 02:49:15.018851 26924 catalog_manager.cc:4841] TS bb3f7bf9227c4d1e9934caa4ae71d6eb (127.7.246.67:39597): tablet 2deb19779d254865ba499708fe266bce (table test-table [id=2723c280b0c04032905676c6c659bcfa]) successfully deleted
I20260619 02:49:15.019232 27427 ts_tablet_manager.cc:1939] T 2deb19779d254865ba499708fe266bce P 3499088de1f44c70b53ea42a8bbfc708: Deleting consensus metadata
I20260619 02:49:15.019320 27120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:15.108048 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26998
I20260619 02:49:15.108258 26925 catalog_manager.cc:4841] TS 3499088de1f44c70b53ea42a8bbfc708 (127.7.246.65:43761): tablet 2deb19779d254865ba499708fe266bce (table test-table [id=2723c280b0c04032905676c6c659bcfa]) successfully deleted
I20260619 02:49:15.132645 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27129
I20260619 02:49:15.140468 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:15.133487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.7.246.66:44451)
0619 02:49:15.133695 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:15.133702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:15.133723 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:49:15.133824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:15.133830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:15.133934 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:15.134037 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:15.134410 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.135497 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.136813 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.137460 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.137642 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:15.138834 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:15.138845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:15.138864 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:15.139627 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:15.139722 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:15.139747 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:15.139846 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:15.139952 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:49:15.140058 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:49:15.141214 27251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:15.228332 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27129
I20260619 02:49:15.250860 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27259
I20260619 02:49:15.258369 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:15.251811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.7.246.67:39597)
0619 02:49:15.252001 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:15.252008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:15.252027 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:49:15.252127 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:15.252139 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:15.252253 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:15.252370 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:15.252798 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.253676 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.254924 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:15.255385 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:15.255534 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:15.256634 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:15.256643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:15.256657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:15.257350 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:15.257410 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:15.257437 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:15.257755 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:15.257882 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:49:15.257991 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:49:15.259110 27381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:15.391379 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27259
I20260619 02:49:15.411324 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 26908
I20260619 02:49:15.412308 26969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:15.519816 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 26908
2026-06-19T02:49:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2730 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:15 dist-test-slave-9tkz krb5kdc[27436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:15 dist-test-slave-9tkz krb5kdc[27436](info): set up 2 sockets
Jun 19 02:49:15 dist-test-slave-9tkz krb5kdc[27436](info): commencing operation
krb5kdc: starting...
W20260619 02:49:16.625886 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:16 dist-test-slave-9tkz krb5kdc[27436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:16.860992 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40287
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:40287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:17.182296 27449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:17.182673 27449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:17.182862 27449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:17.190155 27449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:17.190258 27449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:17.190359 27449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:17.190457 27449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:17.190541 27449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:17.200335 27449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:40287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:17.201638 27449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:17.202968 27449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:17.214962 27457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:17.215018 27454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:17.215035 27455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:17.216017 27449 server_base.cc:1034] running on GCE node
I20260619 02:49:17.216686 27449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:17.218030 27449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:17.219264 27449 hybrid_clock.cc:648] HybridClock initialized: now 1781837357219204 us; error 29 us; skew 500 ppm
I20260619 02:49:17.219702 27449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:17 dist-test-slave-9tkz krb5kdc[27436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:17.225925 27449 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:17.227326 27449 webserver.cc:460] Webserver started at http://127.7.246.126:38493/ using document root <none> and password file <none>
I20260619 02:49:17.227984 27449 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:17.228112 27449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:17.228426 27449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:17.231374 27449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "3f6f7078e21d42feaa97241818287817"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "656c8d8679d6643d713523d93e6b8958"
server_key_iv: "3e419c589b7e7d054cddf4c263e40072"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.232115 27449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "3f6f7078e21d42feaa97241818287817"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "656c8d8679d6643d713523d93e6b8958"
server_key_iv: "3e419c589b7e7d054cddf4c263e40072"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.237071 27449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:17.240443 27464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:17.241818 27449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:17.241988 27449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "3f6f7078e21d42feaa97241818287817"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "656c8d8679d6643d713523d93e6b8958"
server_key_iv: "3e419c589b7e7d054cddf4c263e40072"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.242170 27449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:17.270247 27449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:17.274240 27449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:17.274533 27449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:17.293308 27449 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:40287
I20260619 02:49:17.293303 27515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:40287 every 8 connection(s)
I20260619 02:49:17.294844 27449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:17.298769 27516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:17.304728 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27449
I20260619 02:49:17.304917 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:17.305366 8153 external_mini_cluster.cc:1428] Setting key 4f46a7ac53fc4e175b1f09f31441a372
I20260619 02:49:17.310454 27516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: Bootstrap starting.
I20260619 02:49:17.314554 27516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:17.315927 27516 log.cc:824] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:17.319684 27516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: No bootstrap required, opened a new log
Jun 19 02:49:17 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:17.331274 27516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } }
I20260619 02:49:17.331770 27516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:17.331794 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.307425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.7.246.126:40287)
0619 02:49:17.308341 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:17.308352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:17.308393 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:17.310154 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:17.310176 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:17.310554 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:17.310909 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:17.312933 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.314302 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.316345 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.317475 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.317676 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.322701 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:17.322713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:17.322734 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:17.322780 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:17.327000 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.327696 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.327705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.327713 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.327988 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.328338 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.328348 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.328352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.329142 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:17.329493 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:17.329873 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 02:49:17.330346 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":448,"threads_started":1}
I20260619 02:49:17.331920 27516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6f7078e21d42feaa97241818287817, State: Initialized, Role: FOLLOWER
I20260619 02:49:17.332559 27516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } }
I20260619 02:49:17.332796 27516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:17.332896 27516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:17.333030 27516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:17.335547 27516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } }
I20260619 02:49:17.336092 27516 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6f7078e21d42feaa97241818287817; no voters:
I20260619 02:49:17.336709 27516 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:17.336859 27521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:17.337684 27521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [term 1 LEADER]: Becoming Leader. State: Replica: 3f6f7078e21d42feaa97241818287817, State: Running, Role: LEADER
I20260619 02:49:17.338193 27521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } }
I20260619 02:49:17.339048 27516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:17.340742 27523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6f7078e21d42feaa97241818287817. Latest consensus state: current_term: 1 leader_uuid: "3f6f7078e21d42feaa97241818287817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } } }
I20260619 02:49:17.341485 27523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:17.341485 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6f7078e21d42feaa97241818287817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6f7078e21d42feaa97241818287817" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40287 } } }
I20260619 02:49:17.341861 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:17.342527 27526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:17.347636 27526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:17.359623 27526 catalog_manager.cc:1347] Generated new cluster ID: 4c68f7a74b9d46de846668d9403acc6e
I20260619 02:49:17.360229 27526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:17.376360 27526 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:17.377859 27526 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:17.387392 27526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: Generated new TSK 0
I20260619 02:49:17.388175 27526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.7.246.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
Entry for principal kudu/127.7.246.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab.
I20260619 02:49:17.448205 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:17.770957 27542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:17.771225 27542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:17.771319 27542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:17.778409 27542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:17.778510 27542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:17.778620 27542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:17.788190 27542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.65.keytab
--principal=kudu/127.7.246.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:17.789417 27542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:17.790680 27542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:17.803324 27547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:17.803299 27548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:17.803711 27550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:17.804533 27542 server_base.cc:1034] running on GCE node
I20260619 02:49:17.805109 27542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:17.806080 27542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:17.807277 27542 hybrid_clock.cc:648] HybridClock initialized: now 1781837357807231 us; error 47 us; skew 500 ppm
I20260619 02:49:17.807600 27542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:17 dist-test-slave-9tkz krb5kdc[27436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:17.810618 27542 init.cc:377] Logged in from keytab as kudu/127.7.246.65@KRBTEST.COM (short username kudu)
I20260619 02:49:17.812013 27542 webserver.cc:460] Webserver started at http://127.7.246.65:43927/ using document root <none> and password file <none>
I20260619 02:49:17.812629 27542 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:17.812741 27542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:17.812986 27542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:17.815768 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "cb8b87d8d8714feca95ecef596baf523"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "a5fe61d572b009a823a7bc47726c5909"
server_key_iv: "644b5c054a7b90e4307c58d5c60dbec1"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.816488 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "cb8b87d8d8714feca95ecef596baf523"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "a5fe61d572b009a823a7bc47726c5909"
server_key_iv: "644b5c054a7b90e4307c58d5c60dbec1"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.821396 27542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 02:49:17.824736 27557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:17.826078 27542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:17.826251 27542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "cb8b87d8d8714feca95ecef596baf523"
format_stamp: "Formatted at 2026-06-19 02:49:17 on dist-test-slave-9tkz"
server_key: "a5fe61d572b009a823a7bc47726c5909"
server_key_iv: "644b5c054a7b90e4307c58d5c60dbec1"
server_key_version: "encryptionkey@0"
I20260619 02:49:17.826437 27542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:17.843080 27542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:17.847168 27542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:17.847455 27542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:17.848407 27542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:17.849788 27542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:17.849865 27542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:17.850030 27542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:17.850090 27542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:17.883126 27542 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:37819
I20260619 02:49:17.883263 27669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:37819 every 8 connection(s)
I20260619 02:49:17.884533 27542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:17.891039 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27542
I20260619 02:49:17.891206 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:17.891664 8153 external_mini_cluster.cc:1428] Setting key 8fd44bff589a2382098d966d58467323
Jun 19 02:49:17 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837357, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.65@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:17.905105 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.886964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:39507 (local address 127.7.246.126:40287)
0619 02:49:17.887527 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:17.887536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:17.888331 (+ 795us) server_negotiation.cc:407] Connection header received
0619 02:49:17.890376 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:17.890382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:17.890476 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:17.890603 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:17.892278 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.893045 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.894776 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.895207 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.895345 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.898776 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:17.898782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:17.898787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:17.898831 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:17.901891 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.902813 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.902817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.902819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.902938 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:17.903407 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:17.903411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:17.903413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:17.903676 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:17.903798 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:17.904649 (+ 851us) server_negotiation.cc:299] Negotiation successful
0619 02:49:17.904845 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":329,"threads_started":1}
I20260619 02:49:17.906582 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:17.887504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40287 (local address 127.7.246.65:39507)
0619 02:49:17.888167 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:17.888212 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:17.890038 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:17.890850 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:17.890858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:17.891466 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:17.892079 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:17.892096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.893203 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.893208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:17.894587 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:17.894600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:17.895432 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:17.895442 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:17.895681 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:17.896375 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:17.896397 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:17.898561 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:17.902022 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:17.902034 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:17.902053 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:17.902690 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:17.903039 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:17.903046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:17.903052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:17.903290 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:17.903810 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:17.903821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:17.904265 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 02:49:17.904569 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 02:49:17.905124 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":249,"threads_started":1}
I20260619 02:49:17.908164 27670 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40287
I20260619 02:49:17.908471 27670 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:17.909189 27670 heartbeater.cc:511] Master 127.7.246.126:40287 requested a full tablet report, sending...
I20260619 02:49:17.911340 27481 ts_manager.cc:194] Registered new tserver with Master: cb8b87d8d8714feca95ecef596baf523 (127.7.246.65:37819)
I20260619 02:49:17.912794 27481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.65@KRBTEST.COM'} at 127.7.246.65:39507
WARNING: no policy specified for kudu/127.7.246.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
Entry for principal kudu/127.7.246.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab.
I20260619 02:49:17.948186 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:18.267561 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:18.267865 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:18.268038 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:18.275138 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:18.275309 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:18.275482 27676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:18.285267 27676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.66.keytab
--principal=kudu/127.7.246.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:18.286549 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:18.287827 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:18.300841 27682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:18.301054 27681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:18.301867 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:18.302269 27676 server_base.cc:1034] running on GCE node
I20260619 02:49:18.302852 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:18.303795 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:18.305017 27676 hybrid_clock.cc:648] HybridClock initialized: now 1781837358304970 us; error 42 us; skew 500 ppm
I20260619 02:49:18.305316 27676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:18 dist-test-slave-9tkz krb5kdc[27436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837358, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:18.308521 27676 init.cc:377] Logged in from keytab as kudu/127.7.246.66@KRBTEST.COM (short username kudu)
I20260619 02:49:18.310088 27676 webserver.cc:460] Webserver started at http://127.7.246.66:35163/ using document root <none> and password file <none>
I20260619 02:49:18.310742 27676 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:18.310884 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:18.311199 27676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:18.314007 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "ff803462c3bb461eb1f62196534c8ffd"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "0d8cec715f8a5fba73c3b52b7a32a8e0"
server_key_iv: "957b3d46a2d9c23adb08d04553bbf185"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.314764 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "ff803462c3bb461eb1f62196534c8ffd"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "0d8cec715f8a5fba73c3b52b7a32a8e0"
server_key_iv: "957b3d46a2d9c23adb08d04553bbf185"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.319672 27676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:49:18.323190 27691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.324517 27676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:18.324689 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "ff803462c3bb461eb1f62196534c8ffd"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "0d8cec715f8a5fba73c3b52b7a32a8e0"
server_key_iv: "957b3d46a2d9c23adb08d04553bbf185"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.324891 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:18.343211 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:18.347476 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:18.347774 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:18.348742 27676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:18.350080 27676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:18.350189 27676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.350342 27676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:18.350412 27676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.383020 27676 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:32777
I20260619 02:49:18.383061 27803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:32777 every 8 connection(s)
I20260619 02:49:18.384263 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:18.391928 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27676
I20260619 02:49:18.392112 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:18.392550 8153 external_mini_cluster.cc:1428] Setting key 27a6c65b75a0759059e99f01501882ca
Jun 19 02:49:18 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837358, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.66@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:18.407161 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.386690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:58975 (local address 127.7.246.126:40287)
0619 02:49:18.386868 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.386872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.388220 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 02:49:18.390358 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.390362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.390429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.390507 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:18.392261 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.393154 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.395223 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.396102 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.396281 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.400470 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:18.400475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:18.400479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:18.400509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:18.403359 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.404278 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.404287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.404292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.404463 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.404988 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.404996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.405001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.405405 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:18.405608 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.406456 (+ 848us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.406706 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:49:18.408267 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.387134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40287 (local address 127.7.246.66:58975)
0619 02:49:18.388049 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:18.388095 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:18.390040 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:18.390798 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:18.390807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:18.391373 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:18.392023 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.392075 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.393351 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.393355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.395062 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.395074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.396329 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.396334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.396525 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.397253 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:18.397279 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:18.400253 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:18.403522 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:18.403536 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:18.403557 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:18.404121 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:18.404590 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:18.404593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:18.404596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:18.404851 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:18.405611 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:18.405626 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:18.406015 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 02:49:18.406303 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 02:49:18.406712 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":459,"threads_started":1}
I20260619 02:49:18.409996 27804 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40287
I20260619 02:49:18.410287 27804 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:18.411048 27804 heartbeater.cc:511] Master 127.7.246.126:40287 requested a full tablet report, sending...
I20260619 02:49:18.412458 27481 ts_manager.cc:194] Registered new tserver with Master: ff803462c3bb461eb1f62196534c8ffd (127.7.246.66:32777)
I20260619 02:49:18.413280 27481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.66@KRBTEST.COM'} at 127.7.246.66:58975
WARNING: no policy specified for kudu/127.7.246.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
Entry for principal kudu/127.7.246.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab.
I20260619 02:49:18.448700 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:18.778427 27809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:18.778801 27809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:18.778990 27809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:18.786859 27809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:18.787029 27809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:18.787317 27809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:18.797860 27809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.67.keytab
--principal=kudu/127.7.246.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:40287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:18.799189 27809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:18.800634 27809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:18.814121 27817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:18.814219 27814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:18.814747 27815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:18.815178 27809 server_base.cc:1034] running on GCE node
I20260619 02:49:18.815824 27809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:18.816752 27809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:18.817947 27809 hybrid_clock.cc:648] HybridClock initialized: now 1781837358817902 us; error 36 us; skew 500 ppm
I20260619 02:49:18.818238 27809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:18 dist-test-slave-9tkz krb5kdc[27436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837358, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:18.821522 27809 init.cc:377] Logged in from keytab as kudu/127.7.246.67@KRBTEST.COM (short username kudu)
I20260619 02:49:18.823196 27809 webserver.cc:460] Webserver started at http://127.7.246.67:37575/ using document root <none> and password file <none>
I20260619 02:49:18.823911 27809 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:18.824054 27809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:18.824366 27809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:18.827314 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "c80a8bb8a417a24cecd957f67713f2d3"
server_key_iv: "e557c9651a3e73e10d43a6c5121d5955"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.828111 27809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "c80a8bb8a417a24cecd957f67713f2d3"
server_key_iv: "e557c9651a3e73e10d43a6c5121d5955"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.833459 27809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:18.836860 27824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.838176 27809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:18.838344 27809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0"
format_stamp: "Formatted at 2026-06-19 02:49:18 on dist-test-slave-9tkz"
server_key: "c80a8bb8a417a24cecd957f67713f2d3"
server_key_iv: "e557c9651a3e73e10d43a6c5121d5955"
server_key_version: "encryptionkey@0"
I20260619 02:49:18.838564 27809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:18.858698 27809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:18.862980 27809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:18.863261 27809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:18.864216 27809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:18.865487 27809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:18.865609 27809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.865759 27809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:18.865830 27809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:18.899272 27809 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:38249
I20260619 02:49:18.899355 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:38249 every 8 connection(s)
I20260619 02:49:18.900511 27809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:18.901647 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27809
I20260619 02:49:18.901862 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:18.902418 8153 external_mini_cluster.cc:1428] Setting key e220a1928e3d8866c6f37ddc5d39d8f9
Jun 19 02:49:18 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837358, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.67@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:18.915740 27670 heartbeater.cc:503] Master 127.7.246.126:40287 was elected leader, sending a full tablet report...
I20260619 02:49:18.921139 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.903477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:60421 (local address 127.7.246.126:40287)
0619 02:49:18.903735 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.903742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.904907 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 02:49:18.907213 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.907217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.907278 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.907366 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:18.908896 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.909838 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.911408 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.911800 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.911892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.915434 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:18.915443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:18.915447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:18.915478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:18.918626 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.919286 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.919289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.919291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.919419 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.919814 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.919817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.919819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.920070 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:18.920173 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.920756 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.920908 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:49:18.922377 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.903879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:40287 (local address 127.7.246.67:60421)
0619 02:49:18.904743 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:18.904781 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:18.906962 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:18.907578 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:18.907585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:18.908158 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:18.908750 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.908769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.910011 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.910016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.911278 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.911291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.912006 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.912012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.912228 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.912903 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:18.912933 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:18.915228 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:18.918747 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:18.918753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:18.918766 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:18.919187 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:18.919510 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:18.919513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:18.919516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:18.919692 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:18.920169 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:18.920175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:18.920472 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 02:49:18.920690 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 02:49:18.921080 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
I20260619 02:49:18.923599 27937 heartbeater.cc:347] Connected to a master server at 127.7.246.126:40287
I20260619 02:49:18.923957 27937 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:18.924719 27937 heartbeater.cc:511] Master 127.7.246.126:40287 requested a full tablet report, sending...
I20260619 02:49:18.926163 27481 ts_manager.cc:194] Registered new tserver with Master: fcfabe4ca1ef4043b2ac329fe114b8a0 (127.7.246.67:38249)
I20260619 02:49:18.926904 27481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.7.246.67@KRBTEST.COM'} at 127.7.246.67:60421
I20260619 02:49:18.930732 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:18.947485 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.934731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.7.246.126:40287)
0619 02:49:18.934964 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.934969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.935225 (+ 256us) server_negotiation.cc:407] Connection header received
0619 02:49:18.935404 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.935410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.935518 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.935616 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:18.936905 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.937598 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.938912 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.939349 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.939439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.941884 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:18.941889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:18.941892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:18.941924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:18.944678 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.945436 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.945440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.945443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.945641 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:18.946063 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:18.946066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:18.946068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:18.946452 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:18.946713 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.947012 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.947185 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:49:18.951629 27481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:18.958748 27481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:18.982213 27949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.970990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:37819 (local address 127.0.0.1:38300)
0619 02:49:18.972021 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:18.972073 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:18.972257 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:18.972941 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:18.972952 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:18.972996 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:18.973450 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.973467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.976022 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.976030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.978953 (+ 2923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.978967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.980975 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.980987 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.981218 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.981323 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 02:49:18.981500 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 02:49:18.981726 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":79,"thread_start_us":1035,"threads_started":1}
I20260619 02:49:18.983403 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.971338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.7.246.65:37819)
0619 02:49:18.972090 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.972097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.972152 (+ 55us) server_negotiation.cc:407] Connection header received
0619 02:49:18.972472 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.972478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.972670 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.972926 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:18.973711 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.975787 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.979166 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.980735 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.980903 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.981172 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.981589 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.981773 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":418,"threads_started":1}
I20260619 02:49:18.984881 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.973124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.7.246.67:38249)
0619 02:49:18.973764 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.973775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.975240 (+ 1465us) server_negotiation.cc:407] Connection header received
0619 02:49:18.975632 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.975645 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.975981 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.976203 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:18.978148 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.979628 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.982150 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.983659 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.983751 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.983916 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.984382 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.984535 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":313,"threads_started":1}
I20260619 02:49:18.984964 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.972774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:38249 (local address 127.0.0.1:36872)
0619 02:49:18.973967 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:18.973981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:18.975392 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:18.976549 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:18.976557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:18.976592 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:18.977939 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.977953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.979770 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.979786 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.981925 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.981941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.984067 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.984072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.984185 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.984207 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:49:18.984311 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:49:18.984468 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":761,"thread_start_us":989,"threads_started":1}
I20260619 02:49:18.985862 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.974364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.7.246.66:32777)
0619 02:49:18.977109 (+ 2745us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:18.977114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:18.977630 (+ 516us) server_negotiation.cc:407] Connection header received
0619 02:49:18.977927 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:18.977934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:18.978169 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:18.978324 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:18.979407 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.980534 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.983121 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.984413 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.984544 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.984746 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:18.985294 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 02:49:18.985470 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2600,"thread_start_us":2478,"threads_started":1}
I20260619 02:49:18.986528 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:18.974126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:32777 (local address 127.0.0.1:59116)
0619 02:49:18.977467 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:18.977481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:18.977652 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:18.978620 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:18.978625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:18.978647 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:18.979204 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.979221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.980684 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.980690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.982979 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:18.982991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:18.984910 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:18.984917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:18.985084 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:18.985116 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 02:49:18.986044 (+ 928us) client_negotiation.cc:240] Negotiation successful
0619 02:49:18.986177 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3233,"mutex_wait_us":3,"thread_start_us":3105,"threads_started":1}
I20260619 02:49:18.987529 27605 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a (DEFAULT_TABLE table=test-table [id=d1e056af935347f7860b7a8f93c6bba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:18.988407 27872 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a (DEFAULT_TABLE table=test-table [id=d1e056af935347f7860b7a8f93c6bba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:18.989001 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6f95db0f49477cbb52e4d2a7385d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:18.989494 27872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6f95db0f49477cbb52e4d2a7385d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:18.996258 27739 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a (DEFAULT_TABLE table=test-table [id=d1e056af935347f7860b7a8f93c6bba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:18.997423 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6f95db0f49477cbb52e4d2a7385d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:19.002400 27955 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Bootstrap starting.
I20260619 02:49:19.006295 27956 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Bootstrap starting.
I20260619 02:49:19.006489 27955 tablet_bootstrap.cc:654] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:19.007892 27955 log.cc:824] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:19.009680 27956 tablet_bootstrap.cc:654] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:19.010979 27956 log.cc:824] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:19.011150 27955 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: No bootstrap required, opened a new log
I20260619 02:49:19.011503 27955 ts_tablet_manager.cc:1397] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 02:49:19.011947 27958 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Bootstrap starting.
I20260619 02:49:19.015202 27958 tablet_bootstrap.cc:654] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:19.015410 27956 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: No bootstrap required, opened a new log
I20260619 02:49:19.015668 27956 ts_tablet_manager.cc:1397] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 02:49:19.016301 27958 log.cc:824] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:19.018848 27958 tablet_bootstrap.cc:492] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: No bootstrap required, opened a new log
I20260619 02:49:19.019115 27958 ts_tablet_manager.cc:1397] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 02:49:19.022804 27955 raft_consensus.cc:348] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.023238 27955 raft_consensus.cc:374] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:19.023382 27955 raft_consensus.cc:729] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8b87d8d8714feca95ecef596baf523, State: Initialized, Role: FOLLOWER
I20260619 02:49:19.023991 27955 consensus_queue.cc:260] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.025842 27955 ts_tablet_manager.cc:1428] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260619 02:49:19.026176 27956 raft_consensus.cc:348] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.026636 27956 raft_consensus.cc:374] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:19.026759 27956 raft_consensus.cc:729] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfabe4ca1ef4043b2ac329fe114b8a0, State: Initialized, Role: FOLLOWER
I20260619 02:49:19.027348 27956 consensus_queue.cc:260] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.028556 27937 heartbeater.cc:503] Master 127.7.246.126:40287 was elected leader, sending a full tablet report...
I20260619 02:49:19.029223 27956 ts_tablet_manager.cc:1428] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 02:49:19.029311 27958 raft_consensus.cc:348] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.029811 27958 raft_consensus.cc:374] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:19.029939 27958 raft_consensus.cc:729] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff803462c3bb461eb1f62196534c8ffd, State: Initialized, Role: FOLLOWER
I20260619 02:49:19.030755 27958 consensus_queue.cc:260] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.032152 27804 heartbeater.cc:503] Master 127.7.246.126:40287 was elected leader, sending a full tablet report...
I20260619 02:49:19.032783 27958 ts_tablet_manager.cc:1428] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260619 02:49:19.136345 27805 tablet.cc:2369] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:19.136850 27671 tablet.cc:2369] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:19.152225 27938 tablet.cc:2369] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:19.198098 27962 raft_consensus.cc:482] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:19.198395 27962 raft_consensus.cc:504] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.200080 27962 leader_election.cc:288] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb8b87d8d8714feca95ecef596baf523 (127.7.246.65:37819), ff803462c3bb461eb1f62196534c8ffd (127.7.246.66:32777)
I20260619 02:49:19.206638 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.200433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:37819 (local address 127.7.246.67:53999)
0619 02:49:19.200630 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.200651 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.200778 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.201338 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.201343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.201381 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:19.201611 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.201639 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.202939 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.202944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.204412 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.204423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.205903 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.205909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.206099 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.206136 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:49:19.206226 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:49:19.206363 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:49:19.206727 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.200515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:53999 (local address 127.7.246.65:37819)
0619 02:49:19.200691 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.200695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.200715 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:49:19.200941 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.200948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.201039 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.201144 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:19.201797 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.202807 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.204596 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.205748 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.205908 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.205978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.206381 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.206498 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:49:19.207484 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.201106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:36527 (local address 127.7.246.66:32777)
0619 02:49:19.201308 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.201314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.201660 (+ 346us) server_negotiation.cc:407] Connection header received
0619 02:49:19.201825 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.201829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.201889 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.201958 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:19.202522 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.203440 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.205394 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.206436 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.206566 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.206631 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.207157 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.207284 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:49:19.207569 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.200837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:32777 (local address 127.7.246.67:36527)
0619 02:49:19.201514 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:19.201527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:19.201680 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:19.201997 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:19.202001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:19.202028 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:19.202385 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.202396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.203585 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.203589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.205226 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:19.205241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.206765 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.206770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:19.206952 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.206988 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:49:19.207120 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:49:19.207244 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":404,"threads_started":1}
I20260619 02:49:19.207747 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6f95db0f49477cbb52e4d2a7385d8a" candidate_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b87d8d8714feca95ecef596baf523" is_pre_election: true
I20260619 02:49:19.208273 27625 raft_consensus.cc:2454] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfabe4ca1ef4043b2ac329fe114b8a0 in term 0.
I20260619 02:49:19.208530 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6f95db0f49477cbb52e4d2a7385d8a" candidate_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff803462c3bb461eb1f62196534c8ffd" is_pre_election: true
I20260619 02:49:19.208930 27825 leader_election.cc:302] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb8b87d8d8714feca95ecef596baf523, fcfabe4ca1ef4043b2ac329fe114b8a0; no voters:
I20260619 02:49:19.209025 27759 raft_consensus.cc:2454] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfabe4ca1ef4043b2ac329fe114b8a0 in term 0.
I20260619 02:49:19.209287 27962 raft_consensus.cc:2781] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:19.209438 27962 raft_consensus.cc:482] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:19.209584 27962 raft_consensus.cc:3037] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:19.211779 27962 raft_consensus.cc:504] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.212327 27962 leader_election.cc:288] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [CANDIDATE]: Term 1 election: Requested vote from peers cb8b87d8d8714feca95ecef596baf523 (127.7.246.65:37819), ff803462c3bb461eb1f62196534c8ffd (127.7.246.66:32777)
I20260619 02:49:19.212796 27625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6f95db0f49477cbb52e4d2a7385d8a" candidate_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b87d8d8714feca95ecef596baf523"
I20260619 02:49:19.213016 27625 raft_consensus.cc:3037] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:19.212988 27759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6f95db0f49477cbb52e4d2a7385d8a" candidate_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff803462c3bb461eb1f62196534c8ffd"
I20260619 02:49:19.213183 27759 raft_consensus.cc:3037] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:19.215242 27625 raft_consensus.cc:2454] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfabe4ca1ef4043b2ac329fe114b8a0 in term 1.
I20260619 02:49:19.215432 27759 raft_consensus.cc:2454] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfabe4ca1ef4043b2ac329fe114b8a0 in term 1.
I20260619 02:49:19.215700 27825 leader_election.cc:302] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb8b87d8d8714feca95ecef596baf523, fcfabe4ca1ef4043b2ac329fe114b8a0; no voters:
I20260619 02:49:19.215981 27962 raft_consensus.cc:2781] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:19.216406 27962 raft_consensus.cc:686] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 1 LEADER]: Becoming Leader. State: Replica: fcfabe4ca1ef4043b2ac329fe114b8a0, State: Running, Role: LEADER
I20260619 02:49:19.216858 27962 consensus_queue.cc:237] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } }
I20260619 02:49:19.221292 27480 catalog_manager.cc:5495] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcfabe4ca1ef4043b2ac329fe114b8a0 (127.7.246.67). New cstate: current_term: 1 leader_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfabe4ca1ef4043b2ac329fe114b8a0" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 38249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:19.276006 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.270876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.7.246.67:38249)
0619 02:49:19.271064 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.271068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.271085 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:19.271175 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.271178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.271250 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.271381 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:19.271725 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.272422 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.273978 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.274364 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.274453 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.274965 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:19.275458 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:19.275565 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.275636 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.275695 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:49:19.284298 27759 raft_consensus.cc:1261] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 1 FOLLOWER]: Refusing update from remote peer fcfabe4ca1ef4043b2ac329fe114b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:19.284363 27625 raft_consensus.cc:1261] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 1 FOLLOWER]: Refusing update from remote peer fcfabe4ca1ef4043b2ac329fe114b8a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:19.285290 27965 consensus_queue.cc:1035] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8b87d8d8714feca95ecef596baf523" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:19.285738 27962 consensus_queue.cc:1035] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff803462c3bb461eb1f62196534c8ffd" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:19.298593 27968 mvcc.cc:205] Tried to move back new op lower bound from 7298405823618744320 to 7298405823356325888. Current Snapshot: MvccSnapshot[applied={T|T < 7298405823618744320}]
I20260619 02:49:19.313732 27480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:19.314038 27480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:19.318361 27480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f6f7078e21d42feaa97241818287817: Sending DeleteTablet for 3 replicas of tablet 9a6f95db0f49477cbb52e4d2a7385d8a
I20260619 02:49:19.319872 27739 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:59116
I20260619 02:49:19.319876 27605 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:38300
I20260619 02:49:19.319938 27872 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6f95db0f49477cbb52e4d2a7385d8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:19 UTC) from {username='kudu', principal='kudu/127.7.246.126@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 02:49:19.321309 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27542
I20260619 02:49:19.333485 27977 tablet_replica.cc:330] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: stopping tablet replica
I20260619 02:49:19.334209 27977 raft_consensus.cc:2229] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:19.334988 27977 raft_consensus.cc:2258] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:19.337096 27978 tablet_replica.cc:330] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: stopping tablet replica
I20260619 02:49:19.337575 27978 raft_consensus.cc:2229] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:19.337903 27977 ts_tablet_manager.cc:1905] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:19.338078 27978 raft_consensus.cc:2258] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:19.343907 27977 ts_tablet_manager.cc:1918] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:19.344082 27977 log.cc:1196] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/9a6f95db0f49477cbb52e4d2a7385d8a
I20260619 02:49:19.344589 27977 ts_tablet_manager.cc:1939] T 9a6f95db0f49477cbb52e4d2a7385d8a P ff803462c3bb461eb1f62196534c8ffd: Deleting consensus metadata
I20260619 02:49:19.346503 27465 catalog_manager.cc:4841] TS ff803462c3bb461eb1f62196534c8ffd (127.7.246.66:32777): tablet 9a6f95db0f49477cbb52e4d2a7385d8a (table test-table [id=d1e056af935347f7860b7a8f93c6bba4]) successfully deleted
I20260619 02:49:19.347301 27979 tablet_replica.cc:330] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: stopping tablet replica
I20260619 02:49:19.347738 27979 raft_consensus.cc:2229] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:19.348400 27979 raft_consensus.cc:2258] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:19.350999 27978 ts_tablet_manager.cc:1905] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:19.351101 27979 ts_tablet_manager.cc:1905] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 02:49:19 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.65@KRBTEST.COM
I20260619 02:49:19.358583 27978 ts_tablet_manager.cc:1918] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:19.358821 27978 log.cc:1196] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-0/wal/wals/9a6f95db0f49477cbb52e4d2a7385d8a
I20260619 02:49:19.359315 27979 ts_tablet_manager.cc:1918] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:19.359505 27979 log.cc:1196] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/9a6f95db0f49477cbb52e4d2a7385d8a
I20260619 02:49:19.359535 27978 ts_tablet_manager.cc:1939] T 9a6f95db0f49477cbb52e4d2a7385d8a P cb8b87d8d8714feca95ecef596baf523: Deleting consensus metadata
I20260619 02:49:19.360098 27979 ts_tablet_manager.cc:1939] T 9a6f95db0f49477cbb52e4d2a7385d8a P fcfabe4ca1ef4043b2ac329fe114b8a0: Deleting consensus metadata
I20260619 02:49:19.361642 27465 catalog_manager.cc:4841] TS cb8b87d8d8714feca95ecef596baf523 (127.7.246.65:37819): tablet 9a6f95db0f49477cbb52e4d2a7385d8a (table test-table [id=d1e056af935347f7860b7a8f93c6bba4]) successfully deleted
I20260619 02:49:19.361953 27465 catalog_manager.cc:4841] TS fcfabe4ca1ef4043b2ac329fe114b8a0 (127.7.246.67:38249): tablet 9a6f95db0f49477cbb52e4d2a7385d8a (table test-table [id=d1e056af935347f7860b7a8f93c6bba4]) successfully deleted
I20260619 02:49:19.365899 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.322654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.7.246.65:37819)
0619 02:49:19.322904 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.322914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.346417 (+ 23503us) server_negotiation.cc:407] Connection header received
0619 02:49:19.346889 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.346893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.347020 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.347144 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.348714 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.349936 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.351948 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.352745 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.352861 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.359161 (+ 6300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.359178 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.359196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.359240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.363095 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.363760 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.363771 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.363776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.363937 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.364299 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.364311 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.364314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.364752 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.365050 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.365268 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.365489 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:49:19.366554 27665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:19.472641 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27542
I20260619 02:49:19.490908 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27676
Jun 19 02:49:19 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.66@KRBTEST.COM
I20260619 02:49:19.507807 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.491836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.7.246.66:32777)
0619 02:49:19.492026 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.492031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.492053 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:49:19.492177 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.492183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.492274 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.492395 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.493703 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.494860 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.496102 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.496722 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.496854 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.501093 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.501111 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.501130 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.501175 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.504903 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.505722 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.505733 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.505737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.505893 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.506230 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.506236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.506239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.506726 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.507015 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.507210 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.507445 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:49:19.508549 27799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:19.598467 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27676
I20260619 02:49:19.615506 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27809
Jun 19 02:49:19 dist-test-slave-9tkz krb5kdc[27436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.67@KRBTEST.COM
I20260619 02:49:19.633225 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:19.616314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.7.246.67:38249)
0619 02:49:19.616548 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:19.616554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:19.616576 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:49:19.616647 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:19.616652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:19.616762 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:19.616884 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:19.618225 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.619433 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.620626 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:19.621340 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:19.621486 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:19.626222 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:19.626240 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:19.626260 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:19.626332 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:19.630389 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.631031 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.631042 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.631046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.631260 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:19.631604 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:19.631609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:19.631614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:19.632111 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:19.632431 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:19.632578 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 02:49:19.632811 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 02:49:19.633978 27932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:19.755738 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27809
I20260619 02:49:19.776461 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27449
I20260619 02:49:19.777649 27511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:19.887002 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27449
2026-06-19T02:49:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4369 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T02:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:19Z Disabled control of system clock
I20260619 02:49:19.964872 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41975
--webserver_interface=127.7.246.126
--webserver_port=0
--builtin_ntp_servers=127.7.246.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:20.290925 27988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:20.291260 27988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:20.291407 27988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:20.298616 27988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:20.298739 27988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:20.298806 27988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:20.298911 27988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:20.298988 27988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:20.307927 27988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:41975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:20.309226 27988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:20.311004 27988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:20.322582 27996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.322652 27993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.322688 27994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:20.323503 27988 server_base.cc:1034] running on GCE node
I20260619 02:49:20.324290 27988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:20.325664 27988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:20.326879 27988 hybrid_clock.cc:648] HybridClock initialized: now 1781837360326842 us; error 46 us; skew 500 ppm
I20260619 02:49:20.327301 27988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:20.329629 27988 webserver.cc:460] Webserver started at http://127.7.246.126:44771/ using document root <none> and password file <none>
I20260619 02:49:20.330289 27988 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:20.330418 27988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:20.330724 27988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:20.333422 27988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "c9d6dae101bc4e29bed8eab9e5ded86a"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "2d66074c38ff4b3687a2e6d1531ce057"
server_key_iv: "797978ff44746645a8819e0337e86fc2"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.334247 27988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "c9d6dae101bc4e29bed8eab9e5ded86a"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "2d66074c38ff4b3687a2e6d1531ce057"
server_key_iv: "797978ff44746645a8819e0337e86fc2"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.339113 27988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:49:20.342525 28002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.343811 27988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:49:20.343988 27988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "c9d6dae101bc4e29bed8eab9e5ded86a"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "2d66074c38ff4b3687a2e6d1531ce057"
server_key_iv: "797978ff44746645a8819e0337e86fc2"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.344210 27988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:20.359786 27988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:20.360652 27988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:20.360939 27988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:20.379050 27988 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:41975
I20260619 02:49:20.379067 28053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:41975 every 8 connection(s)
I20260619 02:49:20.380366 27988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:20.383901 28054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:20.388445 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 27988
I20260619 02:49:20.388617 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:20.389171 8153 external_mini_cluster.cc:1428] Setting key 074c2d6612d5611cad88ccfb7936ca7d
I20260619 02:49:20.394279 28054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: Bootstrap starting.
I20260619 02:49:20.397692 28054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:20.398862 28054 log.cc:824] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:20.401961 28054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: No bootstrap required, opened a new log
I20260619 02:49:20.403026 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.391072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.7.246.126:41975)
0619 02:49:20.392098 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.392108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.392148 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:20.393853 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.393875 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.394300 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.394634 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.395225 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.396455 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.398415 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.399113 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.399259 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.400460 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.400472 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.400490 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:20.401146 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.401236 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:20.401259 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.401388 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.401510 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.401840 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":392,"threads_started":1}
I20260619 02:49:20.410100 28054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } }
I20260619 02:49:20.410488 28054 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:20.410621 28054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6dae101bc4e29bed8eab9e5ded86a, State: Initialized, Role: FOLLOWER
I20260619 02:49:20.411185 28054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } }
I20260619 02:49:20.411439 28054 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:20.411559 28054 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:20.411724 28054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:20.413836 28054 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } }
I20260619 02:49:20.414301 28054 leader_election.cc:302] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d6dae101bc4e29bed8eab9e5ded86a; no voters:
I20260619 02:49:20.414877 28054 leader_election.cc:288] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:20.415063 28059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:20.415937 28059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [term 1 LEADER]: Becoming Leader. State: Replica: c9d6dae101bc4e29bed8eab9e5ded86a, State: Running, Role: LEADER
I20260619 02:49:20.416488 28059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } }
I20260619 02:49:20.417017 28054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:20.419212 28061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d6dae101bc4e29bed8eab9e5ded86a. Latest consensus state: current_term: 1 leader_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } } }
I20260619 02:49:20.419620 28061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:20.420256 28060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d6dae101bc4e29bed8eab9e5ded86a" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 41975 } } }
I20260619 02:49:20.420562 28060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:20.425561 28068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:20.430862 28068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:20.439438 28068 catalog_manager.cc:1347] Generated new cluster ID: 29d86f1fa2884a7587b38ccb3d69162e
I20260619 02:49:20.439541 28068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:20.448232 28068 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:20.449645 28068 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:20.454952 28068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: Generated new TSK 0
I20260619 02:49:20.455916 28068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:20.468631 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.65:0
--local_ip_for_outbound_sockets=127.7.246.65
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--builtin_ntp_servers=127.7.246.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:49:20.817360 28079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:20.817742 28079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:20.817907 28079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:20.825289 28079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:20.825429 28079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:20.825664 28079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.65
I20260619 02:49:20.834684 28079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.246.65
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.65
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:20.836004 28079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:20.837873 28079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:20.850710 28087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.850811 28084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:20.851321 28085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:20.852393 28079 server_base.cc:1034] running on GCE node
I20260619 02:49:20.852967 28079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:20.853931 28079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:20.855134 28079 hybrid_clock.cc:648] HybridClock initialized: now 1781837360855076 us; error 56 us; skew 500 ppm
I20260619 02:49:20.855458 28079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:20.857851 28079 webserver.cc:460] Webserver started at http://127.7.246.65:46275/ using document root <none> and password file <none>
I20260619 02:49:20.858532 28079 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:20.858649 28079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:20.858978 28079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:20.861850 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data/instance:
uuid: "dd5fb009fb2840f7af4cfd80e19cc083"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "d1480aab118bce89b304d8e37e1374db"
server_key_iv: "58325d86b5a06e6c22710bdc8e0a8bb1"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.862605 28079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance:
uuid: "dd5fb009fb2840f7af4cfd80e19cc083"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "d1480aab118bce89b304d8e37e1374db"
server_key_iv: "58325d86b5a06e6c22710bdc8e0a8bb1"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.867462 28079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:49:20.870772 28093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.872390 28079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:20.872545 28079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal
uuid: "dd5fb009fb2840f7af4cfd80e19cc083"
format_stamp: "Formatted at 2026-06-19 02:49:20 on dist-test-slave-9tkz"
server_key: "d1480aab118bce89b304d8e37e1374db"
server_key_iv: "58325d86b5a06e6c22710bdc8e0a8bb1"
server_key_version: "encryptionkey@0"
I20260619 02:49:20.872720 28079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:20.904630 28079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:20.905426 28079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:20.905766 28079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:20.906719 28079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:20.908099 28079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:20.908183 28079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.908270 28079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:20.908325 28079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:20.941620 28079 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.65:36831
I20260619 02:49:20.941717 28205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.65:36831 every 8 connection(s)
I20260619 02:49:20.942929 28079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:20.943193 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28079
I20260619 02:49:20.943367 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:20.943859 8153 external_mini_cluster.cc:1428] Setting key fb6220813ba1e4a3992ef2c954395ef1
I20260619 02:49:20.946913 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.66:0
--local_ip_for_outbound_sockets=127.7.246.66
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--builtin_ntp_servers=127.7.246.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:20.956904 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.945630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:49531 (local address 127.7.246.126:41975)
0619 02:49:20.946402 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:20.946408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:20.947617 (+ 1209us) server_negotiation.cc:407] Connection header received
0619 02:49:20.949595 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:20.949603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:20.949695 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:20.949811 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:20.950835 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.952066 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.953635 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.954343 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.954472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.956063 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:20.956068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:20.956071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:20.956089 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:20.956128 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:20.956140 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:20.956236 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:20.956454 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:49:20.956552 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":540,"threads_started":1}
I20260619 02:49:20.958006 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:20.946422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41975 (local address 127.7.246.65:49531)
0619 02:49:20.947461 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:20.947510 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:20.949318 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:20.950002 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:20.950010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:20.950100 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:20.950670 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.950685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.952210 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.952213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.953464 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:20.953475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:20.954586 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:20.954595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:20.954914 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:20.955746 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:20.955749 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:20.955787 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:20.955869 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:20.956235 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:20.956247 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:20.956257 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:20.956444 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 02:49:20.956821 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":326,"threads_started":1}
I20260619 02:49:20.959542 28206 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41975
I20260619 02:49:20.959867 28206 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:20.960702 28206 heartbeater.cc:511] Master 127.7.246.126:41975 requested a full tablet report, sending...
I20260619 02:49:20.962777 28019 ts_manager.cc:194] Registered new tserver with Master: dd5fb009fb2840f7af4cfd80e19cc083 (127.7.246.65:36831)
I20260619 02:49:20.964182 28019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.65:49531
W20260619 02:49:21.267292 28210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:21.267549 28210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:21.267633 28210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:21.274819 28210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:21.274909 28210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:21.275079 28210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.66
I20260619 02:49:21.284363 28210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.246.66
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.66
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:21.285650 28210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:21.287344 28210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:21.300226 28215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:21.300318 28218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:21.300348 28216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:21.301393 28210 server_base.cc:1034] running on GCE node
I20260619 02:49:21.301995 28210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:21.302888 28210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:21.304113 28210 hybrid_clock.cc:648] HybridClock initialized: now 1781837361304067 us; error 59 us; skew 500 ppm
I20260619 02:49:21.304419 28210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:21.306558 28210 webserver.cc:460] Webserver started at http://127.7.246.66:45233/ using document root <none> and password file <none>
I20260619 02:49:21.307265 28210 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:21.307396 28210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:21.307713 28210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:21.310491 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data/instance:
uuid: "65e051c13ea7477b838e596b2d25d3fb"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "2da27341905ab223065144a9da29d4de"
server_key_iv: "fdc80b07d25e8a08da256ed109fdce21"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.311299 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance:
uuid: "65e051c13ea7477b838e596b2d25d3fb"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "2da27341905ab223065144a9da29d4de"
server_key_iv: "fdc80b07d25e8a08da256ed109fdce21"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.316159 28210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:49:21.319725 28224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.320968 28210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:21.321143 28210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal
uuid: "65e051c13ea7477b838e596b2d25d3fb"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "2da27341905ab223065144a9da29d4de"
server_key_iv: "fdc80b07d25e8a08da256ed109fdce21"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.321370 28210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:21.336647 28210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:21.337459 28210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:21.337790 28210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:21.338734 28210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:21.340109 28210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:21.340252 28210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.340374 28210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:21.340462 28210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.379616 28210 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.66:45243
I20260619 02:49:21.379652 28336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.66:45243 every 8 connection(s)
I20260619 02:49:21.380932 28210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:21.388964 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28210
I20260619 02:49:21.389150 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:21.389643 8153 external_mini_cluster.cc:1428] Setting key 0788596bba7098092c7b6e83f003fef4
I20260619 02:49:21.392776 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.246.67:0
--local_ip_for_outbound_sockets=127.7.246.67
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--builtin_ntp_servers=127.7.246.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:49:21.394459 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.383575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.66:45649 (local address 127.7.246.126:41975)
0619 02:49:21.383787 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.383793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.385331 (+ 1538us) server_negotiation.cc:407] Connection header received
0619 02:49:21.387092 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.387098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.387178 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.387293 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:21.388327 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.389466 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.391371 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.391917 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.392051 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.393662 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:21.393667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:21.393670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:21.393687 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:21.393725 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:21.393737 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:21.393786 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.394057 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.394152 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:49:21.395527 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.384092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41975 (local address 127.7.246.66:45649)
0619 02:49:21.385141 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.385189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.386844 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.387488 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.387496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.387577 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:21.388168 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.388183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.389655 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.389662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.391218 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.391231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.392307 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.392318 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.392526 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.393271 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:21.393273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:21.393311 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:21.393383 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:21.393862 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:21.393870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:21.393875 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.394043 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.394419 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260619 02:49:21.396942 28337 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41975
I20260619 02:49:21.397248 28337 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:21.398015 28337 heartbeater.cc:511] Master 127.7.246.126:41975 requested a full tablet report, sending...
I20260619 02:49:21.399384 28019 ts_manager.cc:194] Registered new tserver with Master: 65e051c13ea7477b838e596b2d25d3fb (127.7.246.66:45243)
I20260619 02:49:21.400218 28019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.66:45649
W20260619 02:49:21.713770 28340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:21.714052 28340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:21.714226 28340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:21.721738 28340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:21.721866 28340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:21.722036 28340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.246.67
I20260619 02:49:21.731092 28340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.246.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.246.67
--webserver_port=0
--tserver_master_addrs=127.7.246.126:41975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.246.67
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:21.732323 28340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:21.734108 28340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:21.746784 28345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:21.746824 28346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:21.746827 28348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:21.747457 28340 server_base.cc:1034] running on GCE node
I20260619 02:49:21.748173 28340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:21.749195 28340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:21.750412 28340 hybrid_clock.cc:648] HybridClock initialized: now 1781837361750361 us; error 51 us; skew 500 ppm
I20260619 02:49:21.750739 28340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:49:21.752923 28340 webserver.cc:460] Webserver started at http://127.7.246.67:46705/ using document root <none> and password file <none>
I20260619 02:49:21.753665 28340 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:21.753793 28340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:21.754109 28340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:21.756908 28340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data/instance:
uuid: "f49ddb6c4140456f84e5e64a7108bf20"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "c958f86f019b0d688fef00f125333be2"
server_key_iv: "dfc06b0bbd26cea226d3977e1091e356"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.757730 28340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance:
uuid: "f49ddb6c4140456f84e5e64a7108bf20"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "c958f86f019b0d688fef00f125333be2"
server_key_iv: "dfc06b0bbd26cea226d3977e1091e356"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.762732 28340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:49:21.766980 28354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.768285 28340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:21.768450 28340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal
uuid: "f49ddb6c4140456f84e5e64a7108bf20"
format_stamp: "Formatted at 2026-06-19 02:49:21 on dist-test-slave-9tkz"
server_key: "c958f86f019b0d688fef00f125333be2"
server_key_iv: "dfc06b0bbd26cea226d3977e1091e356"
server_key_version: "encryptionkey@0"
I20260619 02:49:21.768648 28340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:21.782426 28340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:21.783214 28340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:21.783509 28340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:21.784425 28340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:21.785871 28340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:21.785991 28340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.786114 28340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:21.786201 28340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:21.819983 28340 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.67:36447
I20260619 02:49:21.820035 28466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.67:36447 every 8 connection(s)
I20260619 02:49:21.821334 28340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:21.827993 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28340
I20260619 02:49:21.828168 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:21.828675 8153 external_mini_cluster.cc:1428] Setting key e372d2452bb12742a5c52adb0f1911c8
I20260619 02:49:21.834617 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.823899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.67:55279 (local address 127.7.246.126:41975)
0619 02:49:21.824101 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.824106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.825617 (+ 1511us) server_negotiation.cc:407] Connection header received
0619 02:49:21.827352 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.827357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.827425 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.827512 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:21.828931 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.830002 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.831890 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.832287 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.832386 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.833956 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:21.833961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:21.833964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:21.833983 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:21.834014 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:21.834025 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:21.834107 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.834307 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.834379 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:49:21.835536 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.824477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:41975 (local address 127.7.246.67:55279)
0619 02:49:21.825422 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.825463 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.827113 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.827800 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.827814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.827931 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:21.828742 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.828768 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.830165 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.830171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.831754 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.831765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.832471 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.832475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.832654 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.833280 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:21.833282 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:21.833318 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:21.833400 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:21.834118 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:21.834125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:21.834131 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.834266 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.834533 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
I20260619 02:49:21.836817 28467 heartbeater.cc:347] Connected to a master server at 127.7.246.126:41975
I20260619 02:49:21.837093 28467 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:21.837850 28467 heartbeater.cc:511] Master 127.7.246.126:41975 requested a full tablet report, sending...
I20260619 02:49:21.839121 28019 ts_manager.cc:194] Registered new tserver with Master: f49ddb6c4140456f84e5e64a7108bf20 (127.7.246.67:36447)
I20260619 02:49:21.839912 28019 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.246.67:55279
I20260619 02:49:21.843518 8153 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:21.853065 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.847448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.7.246.126:41975)
0619 02:49:21.847622 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.847626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.848071 (+ 445us) server_negotiation.cc:407] Connection header received
0619 02:49:21.848215 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.848219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.848284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.848400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:21.848939 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.849743 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.851072 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.851482 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.851579 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.852509 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:21.852513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:21.852516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:21.852532 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:21.852559 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:21.852570 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:21.852639 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.852758 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.852836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:49:21.856664 28019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:21.863381 28019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:21.890651 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.874575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43458)
0619 02:49:21.876164 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.876197 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.876350 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.878791 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.878801 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.878841 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:21.879275 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.879290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.882917 (+ 3627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.882925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.886979 (+ 4054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.886999 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.889607 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.889619 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.889920 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.889977 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.890106 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.890319 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":1195,"threads_started":1}
I20260619 02:49:21.891687 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.874700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.7.246.65:36831)
0619 02:49:21.876751 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.876759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.876787 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:49:21.876899 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.876909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.877179 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.877369 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:21.879509 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.882723 (+ 3214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.887241 (+ 4518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.889105 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.889267 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.889513 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.891153 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.891323 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1871,"thread_start_us":612,"threads_started":1}
I20260619 02:49:21.895831 28141 tablet_service.cc:1467] Processing CreateTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 (DEFAULT_TABLE table=test-table [id=4380f8ee97004a05ac6ba2479cafecfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:21.897725 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.875205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:36447 (local address 127.0.0.1:36300)
0619 02:49:21.888816 (+ 13611us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.888841 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.888984 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.891517 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.891522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.891551 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:21.891940 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.891958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.893565 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.893573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.895517 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.895530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.896982 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.896988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.897163 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.897199 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.897287 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.897407 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13392,"mutex_wait_us":146,"thread_start_us":14510,"threads_started":1}
I20260619 02:49:21.898124 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.875480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.7.246.67:36447)
0619 02:49:21.890948 (+ 15468us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.890958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.890990 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:21.891088 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.891096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.891328 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.891512 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:21.892137 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.893357 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.895691 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.896749 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.896913 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.897186 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.897667 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.897843 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15284,"thread_start_us":14870,"threads_started":1}
I20260619 02:49:21.898263 28141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51393aa3ca814bac815fc22a3a49ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:21.899595 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.876799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:45243 (local address 127.0.0.1:59038)
0619 02:49:21.889998 (+ 13199us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.890016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.890134 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.891403 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.891408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.891431 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:21.892004 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.892016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.893665 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.893672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.895909 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.895924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.898827 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.898832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.898996 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.899026 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.899120 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.899235 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13041,"mutex_wait_us":48,"thread_start_us":11642,"threads_started":1}
I20260619 02:49:21.899657 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.877369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.7.246.66:45243)
0619 02:49:21.889656 (+ 12287us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.889668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.890130 (+ 462us) server_negotiation.cc:407] Connection header received
0619 02:49:21.890283 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.890295 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.890574 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.891115 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:21.892194 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.893472 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.896114 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.898294 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.898436 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.898669 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.899180 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.899341 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12012,"thread_start_us":11865,"threads_started":1}
I20260619 02:49:21.901417 28402 tablet_service.cc:1467] Processing CreateTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 (DEFAULT_TABLE table=test-table [id=4380f8ee97004a05ac6ba2479cafecfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:21.902657 28402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51393aa3ca814bac815fc22a3a49ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:21.902984 28272 tablet_service.cc:1467] Processing CreateTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 (DEFAULT_TABLE table=test-table [id=4380f8ee97004a05ac6ba2479cafecfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:21.904258 28272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51393aa3ca814bac815fc22a3a49ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:21.914206 28486 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Bootstrap starting.
I20260619 02:49:21.917909 28487 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Bootstrap starting.
I20260619 02:49:21.918792 28486 tablet_bootstrap.cc:654] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:21.919323 28485 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Bootstrap starting.
I20260619 02:49:21.920055 28486 log.cc:824] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:21.921765 28487 tablet_bootstrap.cc:654] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:21.922998 28485 tablet_bootstrap.cc:654] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:21.922979 28487 log.cc:824] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:21.924242 28485 log.cc:824] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:21.924669 28486 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: No bootstrap required, opened a new log
I20260619 02:49:21.924932 28486 ts_tablet_manager.cc:1397] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260619 02:49:21.926301 28487 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: No bootstrap required, opened a new log
I20260619 02:49:21.926602 28487 ts_tablet_manager.cc:1397] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 02:49:21.932118 28485 tablet_bootstrap.cc:492] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: No bootstrap required, opened a new log
I20260619 02:49:21.932376 28485 ts_tablet_manager.cc:1397] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260619 02:49:21.932509 28486 raft_consensus.cc:348] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.932873 28486 raft_consensus.cc:374] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:21.932991 28486 raft_consensus.cc:729] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5fb009fb2840f7af4cfd80e19cc083, State: Initialized, Role: FOLLOWER
I20260619 02:49:21.933631 28486 consensus_queue.cc:260] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.934725 28206 heartbeater.cc:503] Master 127.7.246.126:41975 was elected leader, sending a full tablet report...
I20260619 02:49:21.935232 28486 ts_tablet_manager.cc:1428] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260619 02:49:21.937741 28487 raft_consensus.cc:348] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.938094 28487 raft_consensus.cc:374] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:21.938215 28487 raft_consensus.cc:729] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e051c13ea7477b838e596b2d25d3fb, State: Initialized, Role: FOLLOWER
I20260619 02:49:21.938831 28487 consensus_queue.cc:260] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.940133 28337 heartbeater.cc:503] Master 127.7.246.126:41975 was elected leader, sending a full tablet report...
I20260619 02:49:21.940694 28487 ts_tablet_manager.cc:1428] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 02:49:21.945166 28207 tablet.cc:2369] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:21.951886 28491 raft_consensus.cc:482] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:21.952175 28491 raft_consensus.cc:504] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.953410 28485 raft_consensus.cc:348] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.953835 28485 raft_consensus.cc:374] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:21.953935 28485 raft_consensus.cc:729] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49ddb6c4140456f84e5e64a7108bf20, State: Initialized, Role: FOLLOWER
I20260619 02:49:21.954146 28491 leader_election.cc:288] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65e051c13ea7477b838e596b2d25d3fb (127.7.246.66:45243), f49ddb6c4140456f84e5e64a7108bf20 (127.7.246.67:36447)
I20260619 02:49:21.954756 28485 consensus_queue.cc:260] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.956231 28467 heartbeater.cc:503] Master 127.7.246.126:41975 was elected leader, sending a full tablet report...
I20260619 02:49:21.956979 28485 ts_tablet_manager.cc:1428] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Time spent starting tablet: real 0.024s user 0.013s sys 0.001s
I20260619 02:49:21.962282 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.955141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:52491 (local address 127.7.246.66:45243)
0619 02:49:21.955346 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.955355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.955670 (+ 315us) server_negotiation.cc:407] Connection header received
0619 02:49:21.955899 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.955904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.955982 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.956043 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:21.956930 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.957976 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.960352 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.961304 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.961464 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.961529 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.961865 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.962003 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:49:21.962298 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.954477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.66:45243 (local address 127.7.246.65:52491)
0619 02:49:21.955587 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.955604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.955757 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.956244 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.956249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.956283 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:21.956744 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.956755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.958116 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.958121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.960141 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.960158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.961487 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.961494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.961702 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.961738 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.961832 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.961968 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":7,"thread_start_us":875,"threads_started":1}
I20260619 02:49:21.962589 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.954733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.7.246.65:53909 (local address 127.7.246.67:36447)
0619 02:49:21.955001 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:21.955007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:21.955407 (+ 400us) server_negotiation.cc:407] Connection header received
0619 02:49:21.955725 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:21.955731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:21.955818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:21.955888 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:21.956763 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.957976 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.960450 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.961722 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.961887 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.961949 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:21.962263 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 02:49:21.962395 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 02:49:21.962726 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:21.954432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.67:36447 (local address 127.7.246.65:53909)
0619 02:49:21.955292 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:21.955314 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:21.955525 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:21.956077 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:21.956081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:21.956110 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:21.956548 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.956562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.958178 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.958182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.960190 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:21.960207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:21.961925 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:21.961932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:21.962086 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:21.962122 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:49:21.962221 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:49:21.962376 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"spinlock_wait_cycles":18816,"thread_start_us":548,"threads_started":1}
I20260619 02:49:21.963620 28292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51393aa3ca814bac815fc22a3a49ed09" candidate_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e051c13ea7477b838e596b2d25d3fb" is_pre_election: true
I20260619 02:49:21.963851 28422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51393aa3ca814bac815fc22a3a49ed09" candidate_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49ddb6c4140456f84e5e64a7108bf20" is_pre_election: true
I20260619 02:49:21.964218 28292 raft_consensus.cc:2454] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5fb009fb2840f7af4cfd80e19cc083 in term 0.
I20260619 02:49:21.964336 28422 raft_consensus.cc:2454] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5fb009fb2840f7af4cfd80e19cc083 in term 0.
I20260619 02:49:21.964954 28097 leader_election.cc:302] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 65e051c13ea7477b838e596b2d25d3fb, dd5fb009fb2840f7af4cfd80e19cc083, f49ddb6c4140456f84e5e64a7108bf20; no voters:
I20260619 02:49:21.965387 28491 raft_consensus.cc:2781] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:21.965562 28491 raft_consensus.cc:482] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:21.965693 28491 raft_consensus.cc:3037] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:21.967764 28491 raft_consensus.cc:504] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.968338 28491 leader_election.cc:288] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [CANDIDATE]: Term 1 election: Requested vote from peers 65e051c13ea7477b838e596b2d25d3fb (127.7.246.66:45243), f49ddb6c4140456f84e5e64a7108bf20 (127.7.246.67:36447)
I20260619 02:49:21.968809 28422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51393aa3ca814bac815fc22a3a49ed09" candidate_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49ddb6c4140456f84e5e64a7108bf20"
I20260619 02:49:21.968809 28292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51393aa3ca814bac815fc22a3a49ed09" candidate_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e051c13ea7477b838e596b2d25d3fb"
I20260619 02:49:21.969013 28422 raft_consensus.cc:3037] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:21.969007 28292 raft_consensus.cc:3037] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:21.972265 28292 raft_consensus.cc:2454] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5fb009fb2840f7af4cfd80e19cc083 in term 1.
I20260619 02:49:21.972265 28422 raft_consensus.cc:2454] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5fb009fb2840f7af4cfd80e19cc083 in term 1.
I20260619 02:49:21.972620 28097 leader_election.cc:302] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e051c13ea7477b838e596b2d25d3fb, dd5fb009fb2840f7af4cfd80e19cc083; no voters:
I20260619 02:49:21.972942 28491 raft_consensus.cc:2781] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:21.973387 28491 raft_consensus.cc:686] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [term 1 LEADER]: Becoming Leader. State: Replica: dd5fb009fb2840f7af4cfd80e19cc083, State: Running, Role: LEADER
I20260619 02:49:21.973867 28491 consensus_queue.cc:237] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } }
I20260619 02:49:21.978315 28018 catalog_manager.cc:5495] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd5fb009fb2840f7af4cfd80e19cc083 (127.7.246.65). New cstate: current_term: 1 leader_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd5fb009fb2840f7af4cfd80e19cc083" member_type: VOTER last_known_addr { host: "127.7.246.65" port: 36831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:22.007758 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.002711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.7.246.65:36831)
0619 02:49:22.002960 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.002965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.002982 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:22.003153 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.003158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.003262 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.003389 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:22.003704 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.004403 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.005784 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.006186 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.006274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.006714 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:22.007199 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:22.007303 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.007384 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.007441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 02:49:22.015674 28422 raft_consensus.cc:1261] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 1 FOLLOWER]: Refusing update from remote peer dd5fb009fb2840f7af4cfd80e19cc083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:22.015674 28292 raft_consensus.cc:1261] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 1 FOLLOWER]: Refusing update from remote peer dd5fb009fb2840f7af4cfd80e19cc083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:22.016716 28496 consensus_queue.cc:1035] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49ddb6c4140456f84e5e64a7108bf20" member_type: VOTER last_known_addr { host: "127.7.246.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:22.017130 28491 consensus_queue.cc:1035] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e051c13ea7477b838e596b2d25d3fb" member_type: VOTER last_known_addr { host: "127.7.246.66" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:22.028249 28501 mvcc.cc:205] Tried to move back new op lower bound from 7298405834806353920 to 7298405834648354816. Current Snapshot: MvccSnapshot[applied={T|T < 7298405834806353920}]
I20260619 02:49:22.042979 28018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:22.043251 28018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:22.046842 28018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d6dae101bc4e29bed8eab9e5ded86a: Sending DeleteTablet for 3 replicas of tablet 51393aa3ca814bac815fc22a3a49ed09
I20260619 02:49:22.047912 28141 tablet_service.cc:1514] Processing DeleteTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:22 UTC) from {username='slave'} at 127.0.0.1:43458
I20260619 02:49:22.048188 28402 tablet_service.cc:1514] Processing DeleteTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:22 UTC) from {username='slave'} at 127.0.0.1:36300
I20260619 02:49:22.048200 28272 tablet_service.cc:1514] Processing DeleteTablet for tablet 51393aa3ca814bac815fc22a3a49ed09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:22 UTC) from {username='slave'} at 127.0.0.1:59038
I20260619 02:49:22.049172 28510 tablet_replica.cc:330] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: stopping tablet replica
I20260619 02:49:22.049652 28510 raft_consensus.cc:2229] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:22.049938 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28079
I20260619 02:49:22.050184 28510 raft_consensus.cc:2258] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:22.057900 28510 ts_tablet_manager.cc:1905] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:22.058002 28511 tablet_replica.cc:330] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: stopping tablet replica
I20260619 02:49:22.058516 28511 raft_consensus.cc:2229] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:22.059060 28511 raft_consensus.cc:2258] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:22.061124 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.052848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.7.246.65:36831)
0619 02:49:22.053049 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.053057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.053084 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:49:22.053138 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.053143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.053271 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.053371 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.054011 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.055235 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.057096 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.057871 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.057972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.059451 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.059464 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.059480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:22.060208 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.060268 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:22.060294 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.060376 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.060633 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.060748 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:49:22.061268 28511 ts_tablet_manager.cc:1905] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:22.062412 28201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:22.067143 28510 ts_tablet_manager.cc:1918] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:22.067286 28510 log.cc:1196] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-1/wal/wals/51393aa3ca814bac815fc22a3a49ed09
I20260619 02:49:22.067761 28510 ts_tablet_manager.cc:1939] T 51393aa3ca814bac815fc22a3a49ed09 P 65e051c13ea7477b838e596b2d25d3fb: Deleting consensus metadata
I20260619 02:49:22.069191 28006 catalog_manager.cc:4841] TS 65e051c13ea7477b838e596b2d25d3fb (127.7.246.66:45243): tablet 51393aa3ca814bac815fc22a3a49ed09 (table test-table [id=4380f8ee97004a05ac6ba2479cafecfb]) successfully deleted
I20260619 02:49:22.071235 28511 ts_tablet_manager.cc:1918] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:22.071399 28511 log.cc:1196] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Deleting WAL directory at /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837047186258-8153-0/minicluster-data/ts-2/wal/wals/51393aa3ca814bac815fc22a3a49ed09
I20260619 02:49:22.071808 28511 ts_tablet_manager.cc:1939] T 51393aa3ca814bac815fc22a3a49ed09 P f49ddb6c4140456f84e5e64a7108bf20: Deleting consensus metadata
I20260619 02:49:22.073225 28004 catalog_manager.cc:4841] TS f49ddb6c4140456f84e5e64a7108bf20 (127.7.246.67:36447): tablet 51393aa3ca814bac815fc22a3a49ed09 (table test-table [id=4380f8ee97004a05ac6ba2479cafecfb]) successfully deleted
I20260619 02:49:22.158746 28509 tablet_replica.cc:330] T 51393aa3ca814bac815fc22a3a49ed09 P dd5fb009fb2840f7af4cfd80e19cc083: stopping tablet replica
I20260619 02:49:22.159013 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28079
I20260619 02:49:22.177608 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28210
W20260619 02:49:22.177500 28006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.246.65:36831 (error 108)
I20260619 02:49:22.179508 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.178996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43496)
0619 02:49:22.179188 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.179314 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 02:49:22.179813 28006 catalog_manager.cc:4584] TS dd5fb009fb2840f7af4cfd80e19cc083 (127.7.246.65:36831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51393aa3ca814bac815fc22a3a49ed09: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
I20260619 02:49:22.184614 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.178432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.7.246.66:45243)
0619 02:49:22.178606 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.178611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.178711 (+ 100us) server_negotiation.cc:407] Connection header received
0619 02:49:22.178923 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.178930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.179059 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.179189 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.179644 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.180554 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.181842 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.182279 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.182387 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.183319 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.183327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.183341 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:22.183916 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.183961 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:22.183980 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.184072 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.184161 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.184250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:49:22.185285 28332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:22.233328 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.232860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43508)
0619 02:49:22.233078 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.233164 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 02:49:22.234527 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.234116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43518)
0619 02:49:22.234311 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.234401 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 02:49:22.272271 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28210
I20260619 02:49:22.289881 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28340
I20260619 02:49:22.295583 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.295156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43532)
0619 02:49:22.295357 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.295433 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:49:22.296427 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.296132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43546)
0619 02:49:22.296248 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.296319 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 02:49:22.299031 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.290643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.7.246.67:36447)
0619 02:49:22.290805 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:22.290810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:22.290827 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:49:22.290968 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:22.290971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:22.291055 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:22.291136 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:22.291498 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.292213 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.293360 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:22.293806 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:22.293903 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:22.297767 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:22.297776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:22.297790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:22.298388 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:22.298449 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:22.298468 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:22.298553 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:22.298643 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:49:22.298721 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:49:22.299651 28462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:22.377121 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.376537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43556)
0619 02:49:22.376780 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.376898 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:49:22.378432 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:22.377947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.65:36831 (local address 127.0.0.1:43558)
0619 02:49:22.378176 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:22.378270 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.65:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 02:49:22.388559 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28340
I20260619 02:49:22.405470 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 27988
I20260619 02:49:22.406746 28049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:22.514652 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 27988
2026-06-19T02:49:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2626 ms)
[----------] 6 tests from EncryptionPolicyTest (21383 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:22 dist-test-slave-9tkz krb5kdc[28520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:22 dist-test-slave-9tkz krb5kdc[28520](info): set up 2 sockets
Jun 19 02:49:22 dist-test-slave-9tkz krb5kdc[28520](info): commencing operation
krb5kdc: starting...
W20260619 02:49:23.626178 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:23 dist-test-slave-9tkz krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:23.871227 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:24.199103 28533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:24.199443 28533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:24.199599 28533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:24.206866 28533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:24.207005 28533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:24.207110 28533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:24.207221 28533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:24.207301 28533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:24.216743 28533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:24.218027 28533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:24.219305 28533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:24.230587 28538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:24.230670 28541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:24.230618 28539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:24.231501 28533 server_base.cc:1034] running on GCE node
I20260619 02:49:24.232227 28533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:24.233466 28533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:24.234712 28533 hybrid_clock.cc:648] HybridClock initialized: now 1781837364234646 us; error 36 us; skew 500 ppm
I20260619 02:49:24.235059 28533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:24 dist-test-slave-9tkz krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:24.238306 28533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:24.239908 28533 webserver.cc:460] Webserver started at http://127.0.0.1:42827/ using document root <none> and password file <none>
I20260619 02:49:24.240609 28533 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:24.240778 28533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:24.241103 28533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:24.244321 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "653e182d74864f79b374ae4c87dbcada"
format_stamp: "Formatted at 2026-06-19 02:49:24 on dist-test-slave-9tkz"
server_key: "65dc8ca833015abdca3c057aa3947ab5"
server_key_iv: "d4c6646c452d4c9977e92858483eab14"
server_key_version: "encryptionkey@0"
I20260619 02:49:24.245126 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "653e182d74864f79b374ae4c87dbcada"
format_stamp: "Formatted at 2026-06-19 02:49:24 on dist-test-slave-9tkz"
server_key: "65dc8ca833015abdca3c057aa3947ab5"
server_key_iv: "d4c6646c452d4c9977e92858483eab14"
server_key_version: "encryptionkey@0"
I20260619 02:49:24.250383 28533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:49:24.253681 28548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:24.254987 28533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:49:24.255182 28533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "653e182d74864f79b374ae4c87dbcada"
format_stamp: "Formatted at 2026-06-19 02:49:24 on dist-test-slave-9tkz"
server_key: "65dc8ca833015abdca3c057aa3947ab5"
server_key_iv: "d4c6646c452d4c9977e92858483eab14"
server_key_version: "encryptionkey@0"
I20260619 02:49:24.255379 28533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:24.270915 28533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:24.274977 28533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:24.275202 28533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:24.292541 28533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45561
I20260619 02:49:24.292582 28599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45561 every 8 connection(s)
I20260619 02:49:24.293888 28533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:24.294993 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28533
I20260619 02:49:24.295212 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:24.295827 8153 external_mini_cluster.cc:1428] Setting key 4ff6a682192b7097e0162f5089be509f
I20260619 02:49:24.298241 28600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:49:24 dist-test-slave-9tkz krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:49:24.312407 28600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada: Bootstrap starting.
I20260619 02:49:24.316537 28600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:24.317853 28600 log.cc:824] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:24.320976 28600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada: No bootstrap required, opened a new log
I20260619 02:49:24.321714 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:24.297780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45073 (local address 127.0.0.1:45561)
0619 02:49:24.298544 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:24.298564 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:24.298612 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:24.300958 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:24.300984 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:24.301500 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:24.301963 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:24.304298 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.305760 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.307467 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.308170 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.308316 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:24.313154 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:24.313165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:24.313182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:24.313229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:24.317269 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.318095 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.318101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.318107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.318273 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.318873 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.318879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.318885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.319457 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:24.319662 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:24.320258 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 02:49:24.320538 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":301,"threads_started":1}
I20260619 02:49:24.325487 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:24.329020 28600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 02:49:24.329391 28600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:24.329494 28600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653e182d74864f79b374ae4c87dbcada, State: Initialized, Role: FOLLOWER
I20260619 02:49:24.330247 28600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 02:49:24.330567 28600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:24.330706 28600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:24.330897 28600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:24.333029 28600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 02:49:24.333528 28600 leader_election.cc:302] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 653e182d74864f79b374ae4c87dbcada; no voters:
I20260619 02:49:24.334281 28600 leader_election.cc:288] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:24.334390 28610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:24.336759 28600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:24.337878 28610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [term 1 LEADER]: Becoming Leader. State: Replica: 653e182d74864f79b374ae4c87dbcada, State: Running, Role: LEADER
I20260619 02:49:24.338752 28610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 02:49:24.342267 28611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "653e182d74864f79b374ae4c87dbcada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } } }
I20260619 02:49:24.342717 28611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:24.343194 28612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653e182d74864f79b374ae4c87dbcada. Latest consensus state: current_term: 1 leader_uuid: "653e182d74864f79b374ae4c87dbcada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653e182d74864f79b374ae4c87dbcada" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } } }
I20260619 02:49:24.343528 28612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:24.347345 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:24.330130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41763 (local address 127.0.0.1:45561)
0619 02:49:24.330296 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:24.330300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:24.330809 (+ 509us) server_negotiation.cc:407] Connection header received
0619 02:49:24.331005 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:24.331010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:24.331078 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:24.331178 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:24.332527 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.333350 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.334855 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:24.335495 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:24.335607 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:24.338283 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:24.338289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:24.338294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:24.338331 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:24.342321 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.343409 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.343414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.343418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.343588 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:24.344285 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:24.344293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:24.344299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:24.344753 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:24.345097 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:24.346578 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0619 02:49:24.346795 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:49:24.351529 28621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:24.356884 28621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:24.365418 28621 catalog_manager.cc:1347] Generated new cluster ID: 77dced4f0e1d40cab999be4c5c76000d
I20260619 02:49:24.365558 28621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:24.381407 28621 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:24.382629 28621 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:24.391489 28621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 653e182d74864f79b374ae4c87dbcada: Generated new TSK 0
I20260619 02:49:24.392246 28621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:24.401026 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28533
I20260619 02:49:24.402313 28595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:24.491351 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28533
W20260619 02:49:24.508527 28518 connection.cc:535] client connection to 127.0.0.1:45561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45561): Connection reset by peer (error 104)
2026-06-19T02:49:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:24 dist-test-slave-9tkz krb5kdc[28632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:24 dist-test-slave-9tkz krb5kdc[28632](info): set up 2 sockets
Jun 19 02:49:24 dist-test-slave-9tkz krb5kdc[28632](info): commencing operation
krb5kdc: starting...
W20260619 02:49:25.598111 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:25 dist-test-slave-9tkz krb5kdc[28632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:25.840042 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:26.158546 28645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:26.158828 28645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:26.158994 28645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:26.166108 28645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:26.166213 28645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:26.166290 28645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:26.166381 28645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:26.166491 28645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:26.176100 28645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:26.177349 28645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:26.178680 28645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:26.190835 28650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:26.190908 28651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:26.190881 28653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:26.192030 28645 server_base.cc:1034] running on GCE node
I20260619 02:49:26.192801 28645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:26.194208 28645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:26.195415 28645 hybrid_clock.cc:648] HybridClock initialized: now 1781837366195355 us; error 53 us; skew 500 ppm
I20260619 02:49:26.195793 28645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:26 dist-test-slave-9tkz krb5kdc[28632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:26.198900 28645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:26.200280 28645 webserver.cc:460] Webserver started at http://127.0.0.1:33617/ using document root <none> and password file <none>
I20260619 02:49:26.200923 28645 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:26.201018 28645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:26.201254 28645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:26.204241 28645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "6a5a01e16993446ca6e019a358e2fd6b"
format_stamp: "Formatted at 2026-06-19 02:49:26 on dist-test-slave-9tkz"
server_key: "a4917367e3de7a0e556233f2be932711"
server_key_iv: "8b006a939926914eb9fdea8dcf39149f"
server_key_version: "encryptionkey@0"
I20260619 02:49:26.204921 28645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "6a5a01e16993446ca6e019a358e2fd6b"
format_stamp: "Formatted at 2026-06-19 02:49:26 on dist-test-slave-9tkz"
server_key: "a4917367e3de7a0e556233f2be932711"
server_key_iv: "8b006a939926914eb9fdea8dcf39149f"
server_key_version: "encryptionkey@0"
I20260619 02:49:26.209800 28645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 02:49:26.213157 28660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:26.214497 28645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:26.214695 28645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "6a5a01e16993446ca6e019a358e2fd6b"
format_stamp: "Formatted at 2026-06-19 02:49:26 on dist-test-slave-9tkz"
server_key: "a4917367e3de7a0e556233f2be932711"
server_key_iv: "8b006a939926914eb9fdea8dcf39149f"
server_key_version: "encryptionkey@0"
I20260619 02:49:26.214895 28645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:26.234050 28645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:26.237905 28645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:26.238183 28645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:26.256294 28645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36195
I20260619 02:49:26.256326 28711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36195 every 8 connection(s)
I20260619 02:49:26.257645 28645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:26.261319 28712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:26.262888 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28645
I20260619 02:49:26.263064 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:26.263602 8153 external_mini_cluster.cc:1428] Setting key 8ebb594dc9f450247f4819d894b90d3b
I20260619 02:49:26.275091 28712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b: Bootstrap starting.
I20260619 02:49:26.276641 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:26.265345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53031 (local address 127.0.0.1:36195)
0619 02:49:26.266174 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:26.266185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:26.266228 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:49:26.267801 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:26.267823 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:26.268211 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:26.268550 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:26.269089 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:26.270448 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:26.272325 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:26.273045 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:26.273169 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:26.274419 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:26.274426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:26.274438 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:26.274981 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:26.275051 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:26.275074 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:26.275233 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:26.275478 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:49:26.275677 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":431,"threads_started":1}
I20260619 02:49:26.278234 28712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:26.279261 28712 log.cc:824] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:26.280922 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:26.282269 28712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b: No bootstrap required, opened a new log
I20260619 02:49:26.291446 28712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260619 02:49:26.291862 28712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:26.291988 28712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5a01e16993446ca6e019a358e2fd6b, State: Initialized, Role: FOLLOWER
I20260619 02:49:26.292058 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:26.285205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60637 (local address 127.0.0.1:36195)
0619 02:49:26.285371 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:26.285376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:26.285870 (+ 494us) server_negotiation.cc:407] Connection header received
0619 02:49:26.286042 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:26.286046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:26.286101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:26.286172 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:26.286740 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:26.287509 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:26.289356 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:26.289911 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:26.289997 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:26.291229 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:26.291234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:26.291237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:26.291253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:26.291286 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:26.291300 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:26.291425 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:26.291610 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:49:26.291662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:49:26.292644 28712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260619 02:49:26.292937 28712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:26.293082 28712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:26.293255 28712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:26.295495 28712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260619 02:49:26.295961 28712 leader_election.cc:302] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a5a01e16993446ca6e019a358e2fd6b; no voters:
I20260619 02:49:26.296480 28712 leader_election.cc:288] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:26.296603 28722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:26.297353 28722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [term 1 LEADER]: Becoming Leader. State: Replica: 6a5a01e16993446ca6e019a358e2fd6b, State: Running, Role: LEADER
I20260619 02:49:26.297883 28722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260619 02:49:26.298563 28712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:26.300849 28724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a5a01e16993446ca6e019a358e2fd6b. Latest consensus state: current_term: 1 leader_uuid: "6a5a01e16993446ca6e019a358e2fd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260619 02:49:26.300824 28723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a5a01e16993446ca6e019a358e2fd6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5a01e16993446ca6e019a358e2fd6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260619 02:49:26.301270 28724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:26.301267 28723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:26.303014 28729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:26.308187 28729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:26.317363 28729 catalog_manager.cc:1347] Generated new cluster ID: 8071b4e5c8d24c409b2c3763b50b260c
I20260619 02:49:26.317500 28729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:26.333796 28729 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:26.335031 28729 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:26.345736 28729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a5a01e16993446ca6e019a358e2fd6b: Generated new TSK 0
I20260619 02:49:26.346558 28729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:26.352548 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28645
W20260619 02:49:26.353677 28707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53031
W20260619 02:49:26.354367 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:26.354539 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28645
W20260619 02:49:26.371089 28630 connection.cc:535] client connection to 127.0.0.1:36195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36195): Connection reset by peer (error 104)
2026-06-19T02:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:26 dist-test-slave-9tkz krb5kdc[28743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:26 dist-test-slave-9tkz krb5kdc[28743](info): set up 2 sockets
Jun 19 02:49:26 dist-test-slave-9tkz krb5kdc[28743](info): commencing operation
krb5kdc: starting...
W20260619 02:49:27.446086 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:27 dist-test-slave-9tkz krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:27.693342 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:28.018455 28756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:28.018730 28756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:28.018888 28756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:28.026224 28756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:28.026330 28756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:28.026439 28756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:28.026551 28756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:28.026635 28756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:28.036535 28756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:28.037822 28756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:28.039116 28756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:28.050797 28762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.051024 28764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:28.051451 28761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:28.051847 28756 server_base.cc:1034] running on GCE node
I20260619 02:49:28.052632 28756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:28.054021 28756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:28.055245 28756 hybrid_clock.cc:648] HybridClock initialized: now 1781837368055195 us; error 31 us; skew 500 ppm
I20260619 02:49:28.055590 28756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:28 dist-test-slave-9tkz krb5kdc[28743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:28.064266 28756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:28.065763 28756 webserver.cc:460] Webserver started at http://127.0.0.1:38151/ using document root <none> and password file <none>
I20260619 02:49:28.066542 28756 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:28.066669 28756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:28.066973 28756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:28.070065 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "3f34475d2fea44808140462928d18ed2"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-9tkz"
server_key: "45bb52338ceee2e7263babff21d61277"
server_key_iv: "ec2440e131a4f90abf60e9f5b37c4f2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.070827 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "3f34475d2fea44808140462928d18ed2"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-9tkz"
server_key: "45bb52338ceee2e7263babff21d61277"
server_key_iv: "ec2440e131a4f90abf60e9f5b37c4f2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.075868 28756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 02:49:28.079334 28771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:28.080677 28756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:28.080895 28756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "3f34475d2fea44808140462928d18ed2"
format_stamp: "Formatted at 2026-06-19 02:49:28 on dist-test-slave-9tkz"
server_key: "45bb52338ceee2e7263babff21d61277"
server_key_iv: "ec2440e131a4f90abf60e9f5b37c4f2d"
server_key_version: "encryptionkey@0"
I20260619 02:49:28.081086 28756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:28.086815 28756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:28.090867 28756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:28.091158 28756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:28.109110 28756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34123
I20260619 02:49:28.109149 28822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34123 every 8 connection(s)
I20260619 02:49:28.110638 28756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:28.114305 28823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:28.117153 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28756
I20260619 02:49:28.117341 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:28.117848 8153 external_mini_cluster.cc:1428] Setting key 6f917819a6c4c8cd0c1181d50bfc385d
I20260619 02:49:28.125860 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.119843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48069 (local address 127.0.0.1:34123)
0619 02:49:28.120660 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.120672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.120716 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:49:28.122469 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.122492 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.122846 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.123161 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.124231 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.124257 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.124275 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:28.124296 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.124351 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:28.124373 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.124559 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.124664 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.124864 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":392,"threads_started":1}
I20260619 02:49:28.126572 28823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2: Bootstrap starting.
I20260619 02:49:28.129372 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:28.129956 28823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:28.130920 28823 log.cc:824] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:28.133720 28823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2: No bootstrap required, opened a new log
I20260619 02:49:28.136565 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:28.134070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41663 (local address 127.0.0.1:34123)
0619 02:49:28.134313 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:28.134319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:28.134653 (+ 334us) server_negotiation.cc:407] Connection header received
0619 02:49:28.134832 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:28.134836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:28.134880 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:28.134951 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:28.135876 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:28.135881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:28.135884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:28.135899 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:28.135942 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:28.135957 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:28.136060 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:28.136180 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:49:28.136243 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:49:28.142778 28823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260619 02:49:28.143167 28823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:28.143296 28823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f34475d2fea44808140462928d18ed2, State: Initialized, Role: FOLLOWER
I20260619 02:49:28.143851 28823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260619 02:49:28.144080 28823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:28.144239 28823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:28.144414 28823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:28.146795 28823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260619 02:49:28.147249 28823 leader_election.cc:302] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f34475d2fea44808140462928d18ed2; no voters:
I20260619 02:49:28.147831 28823 leader_election.cc:288] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:28.147990 28833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:28.149098 28833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 3f34475d2fea44808140462928d18ed2, State: Running, Role: LEADER
I20260619 02:49:28.149677 28833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } }
I20260619 02:49:28.150074 28823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:28.152374 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f34475d2fea44808140462928d18ed2. Latest consensus state: current_term: 1 leader_uuid: "3f34475d2fea44808140462928d18ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } } }
I20260619 02:49:28.152791 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:28.152545 28834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f34475d2fea44808140462928d18ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f34475d2fea44808140462928d18ed2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34123 } } }
I20260619 02:49:28.152945 28834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:28.153810 28841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:28.165933 28841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:28.175269 28841 catalog_manager.cc:1347] Generated new cluster ID: e3113a1d9be3449488283d7d060d329f
I20260619 02:49:28.175405 28841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:28.197372 28841 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:28.197916 28841 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:28.225073 28841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f34475d2fea44808140462928d18ed2: Generated new TSK 0
I20260619 02:49:28.226003 28841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:28.237712 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28756
W20260619 02:49:28.238821 28818 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48069
W20260619 02:49:28.239531 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:28.239681 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28756
2026-06-19T02:49:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:28 dist-test-slave-9tkz krb5kdc[28854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:28 dist-test-slave-9tkz krb5kdc[28854](info): set up 2 sockets
Jun 19 02:49:28 dist-test-slave-9tkz krb5kdc[28854](info): commencing operation
krb5kdc: starting...
W20260619 02:49:29.346079 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:29 dist-test-slave-9tkz krb5kdc[28854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:29.594907 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:29.924088 28867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:29.924408 28867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:29.924618 28867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:29.931943 28867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:29.932053 28867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:29.932180 28867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:29.932300 28867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:29.932385 28867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:29.942456 28867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:29.943774 28867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:29.945127 28867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:29.957808 28872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:29.957851 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:29.957882 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:29.958811 28867 server_base.cc:1034] running on GCE node
I20260619 02:49:29.959604 28867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:29.961102 28867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:29.962304 28867 hybrid_clock.cc:648] HybridClock initialized: now 1781837369962255 us; error 34 us; skew 500 ppm
I20260619 02:49:29.962663 28867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:29 dist-test-slave-9tkz krb5kdc[28854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:29.965754 28867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:29.967188 28867 webserver.cc:460] Webserver started at http://127.0.0.1:42515/ using document root <none> and password file <none>
I20260619 02:49:29.967918 28867 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:29.968063 28867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:29.968369 28867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:29.971392 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "122380198d4a47468ac5842f4a46f600"
format_stamp: "Formatted at 2026-06-19 02:49:29 on dist-test-slave-9tkz"
server_key: "8607e32859803081a41d8344d0a38018"
server_key_iv: "991fcae85efd5937d3437604b36acc1a"
server_key_version: "encryptionkey@0"
I20260619 02:49:29.972151 28867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "122380198d4a47468ac5842f4a46f600"
format_stamp: "Formatted at 2026-06-19 02:49:29 on dist-test-slave-9tkz"
server_key: "8607e32859803081a41d8344d0a38018"
server_key_iv: "991fcae85efd5937d3437604b36acc1a"
server_key_version: "encryptionkey@0"
I20260619 02:49:29.977244 28867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:29.980541 28882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:29.982223 28867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:29.982352 28867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "122380198d4a47468ac5842f4a46f600"
format_stamp: "Formatted at 2026-06-19 02:49:29 on dist-test-slave-9tkz"
server_key: "8607e32859803081a41d8344d0a38018"
server_key_iv: "991fcae85efd5937d3437604b36acc1a"
server_key_version: "encryptionkey@0"
I20260619 02:49:29.982517 28867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:30.016284 28867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:30.020403 28867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:30.020640 28867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:30.039072 28867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41689
I20260619 02:49:30.039149 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41689 every 8 connection(s)
I20260619 02:49:30.040478 28867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:30.044274 28934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:30.048480 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28867
I20260619 02:49:30.048651 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:30.049078 8153 external_mini_cluster.cc:1428] Setting key ac2dc90273aa1aab8e37a96efa89aa32
I20260619 02:49:30.055725 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600: Bootstrap starting.
I20260619 02:49:30.059482 28934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:30.060704 28934 log.cc:824] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:30.064254 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600: No bootstrap required, opened a new log
Jun 19 02:49:30 dist-test-slave-9tkz krb5kdc[28854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:49:30.072616 28934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } }
I20260619 02:49:30.073072 28934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:30.073213 28934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122380198d4a47468ac5842f4a46f600, State: Initialized, Role: FOLLOWER
I20260619 02:49:30.073446 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:30.050890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34355 (local address 127.0.0.1:41689)
0619 02:49:30.051654 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:30.051666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:30.051704 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:49:30.053975 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:30.053997 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:30.054342 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:30.054649 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:30.056656 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.058012 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.059919 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.060584 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.060717 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:30.065835 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:30.065843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:30.065870 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:30.065908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:30.069052 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.069884 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.069890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.069897 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.070053 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.070478 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.070482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.070486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.071050 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:30.071398 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:30.071823 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 02:49:30.072227 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
I20260619 02:49:30.074018 28934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } }
I20260619 02:49:30.074363 28934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:30.074533 28934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:30.074743 28934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:30.077250 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:30.077687 28934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } }
I20260619 02:49:30.078193 28934 leader_election.cc:302] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122380198d4a47468ac5842f4a46f600; no voters:
I20260619 02:49:30.078819 28934 leader_election.cc:288] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:30.078954 28939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:30.080217 28939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [term 1 LEADER]: Becoming Leader. State: Replica: 122380198d4a47468ac5842f4a46f600, State: Running, Role: LEADER
I20260619 02:49:30.080906 28939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } }
I20260619 02:49:30.081189 28934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:30.086143 28940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "122380198d4a47468ac5842f4a46f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } } }
I20260619 02:49:30.086540 28940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:30.086248 28941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122380198d4a47468ac5842f4a46f600. Latest consensus state: current_term: 1 leader_uuid: "122380198d4a47468ac5842f4a46f600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122380198d4a47468ac5842f4a46f600" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41689 } } }
I20260619 02:49:30.086666 28941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:30.088217 28952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:30.096388 28952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:30.106025 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:30.083587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33409 (local address 127.0.0.1:41689)
0619 02:49:30.083969 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:30.083979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:30.084587 (+ 608us) server_negotiation.cc:407] Connection header received
0619 02:49:30.088227 (+ 3640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:30.088236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:30.088328 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:30.088455 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:30.090956 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.092492 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.094396 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:30.095100 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:30.095224 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:30.098532 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:30.098540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:30.098545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:30.098583 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:30.102269 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.103126 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.103131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.103134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.103994 (+ 860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:30.104482 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:30.104487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:30.104491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:30.104800 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:30.105048 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:30.105388 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 02:49:30.105633 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260619 02:49:30.111872 28952 catalog_manager.cc:1347] Generated new cluster ID: 433ed531201b456886aac965afc3c698
I20260619 02:49:30.111994 28952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:30.130081 28952 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:30.131428 28952 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:30.138279 28952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 122380198d4a47468ac5842f4a46f600: Generated new TSK 0
I20260619 02:49:30.139094 28952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:30.150678 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28867
I20260619 02:49:30.151611 28929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:30.241698 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28867
2026-06-19T02:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:30 dist-test-slave-9tkz krb5kdc[28966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:30 dist-test-slave-9tkz krb5kdc[28966](info): set up 2 sockets
Jun 19 02:49:30 dist-test-slave-9tkz krb5kdc[28966](info): commencing operation
krb5kdc: starting...
W20260619 02:49:31.373818 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:31 dist-test-slave-9tkz krb5kdc[28966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:31.623121 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:31.946705 28979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:31.946998 28979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:31.947163 28979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:31.954360 28979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:31.954493 28979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:31.954591 28979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:31.954691 28979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:31.954775 28979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:31.964716 28979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:31.966099 28979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:31.967404 28979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:31.979184 28987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.979277 28985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:31.979424 28984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:31.980021 28979 server_base.cc:1034] running on GCE node
I20260619 02:49:31.980751 28979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:31.982100 28979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:31.983299 28979 hybrid_clock.cc:648] HybridClock initialized: now 1781837371983263 us; error 33 us; skew 500 ppm
I20260619 02:49:31.983639 28979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:31 dist-test-slave-9tkz krb5kdc[28966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:31.986600 28979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:31.988018 28979 webserver.cc:460] Webserver started at http://127.0.0.1:37713/ using document root <none> and password file <none>
I20260619 02:49:31.988687 28979 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:31.988833 28979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:31.989122 28979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:31.992046 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "474368a7d10c4197bd2c7543d9423970"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-9tkz"
server_key: "d2578f669bc79e1b02d54eda1b2eec8a"
server_key_iv: "5d8cb19b58de5216d5f2d8b8cf0e3822"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.992882 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "474368a7d10c4197bd2c7543d9423970"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-9tkz"
server_key: "d2578f669bc79e1b02d54eda1b2eec8a"
server_key_iv: "5d8cb19b58de5216d5f2d8b8cf0e3822"
server_key_version: "encryptionkey@0"
I20260619 02:49:31.997893 28979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:32.001152 28994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:32.002420 28979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:32.002612 28979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "474368a7d10c4197bd2c7543d9423970"
format_stamp: "Formatted at 2026-06-19 02:49:31 on dist-test-slave-9tkz"
server_key: "d2578f669bc79e1b02d54eda1b2eec8a"
server_key_iv: "5d8cb19b58de5216d5f2d8b8cf0e3822"
server_key_version: "encryptionkey@0"
I20260619 02:49:32.002811 28979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:32.028760 28979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:32.032586 28979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:32.032874 28979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:32.051038 28979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39105
I20260619 02:49:32.051076 29045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39105 every 8 connection(s)
I20260619 02:49:32.052361 28979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:32.055698 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 28979
I20260619 02:49:32.055886 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:32.056051 29046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:32.056442 8153 external_mini_cluster.cc:1428] Setting key f87da54cb1edb43128ff64f03104c6a0
I20260619 02:49:32.069521 29046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970: Bootstrap starting.
I20260619 02:49:32.070034 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:32.058099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43411 (local address 127.0.0.1:39105)
0619 02:49:32.058936 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:32.058946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:32.058988 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:49:32.060864 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:32.060887 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:32.061279 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:32.061694 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:32.062217 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.063637 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.065405 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.066202 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.066355 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:32.067584 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:32.067592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:32.067609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:32.068176 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:32.068253 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:32.068279 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:32.068377 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:32.068612 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:49:32.068942 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":382,"threads_started":1}
I20260619 02:49:32.072578 29046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:32.073750 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:32.073750 29046 log.cc:824] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:32.076418 29046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970: No bootstrap required, opened a new log
I20260619 02:49:32.085239 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:32.078636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36151 (local address 127.0.0.1:39105)
0619 02:49:32.078828 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:32.078834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:32.079374 (+ 540us) server_negotiation.cc:407] Connection header received
0619 02:49:32.079540 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:32.079544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:32.079606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:32.079691 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:32.080254 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.080994 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.082817 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:32.083350 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:32.083462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:32.084532 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:32.084537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:32.084540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:32.084556 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:32.084591 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:32.084605 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:32.084677 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:32.084819 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:49:32.084885 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:49:32.086050 29046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } }
I20260619 02:49:32.086417 29046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:32.086524 29046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474368a7d10c4197bd2c7543d9423970, State: Initialized, Role: FOLLOWER
I20260619 02:49:32.087045 29046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } }
I20260619 02:49:32.087236 29046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:32.087330 29046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:32.087451 29046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:32.089404 29046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } }
I20260619 02:49:32.089908 29046 leader_election.cc:302] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474368a7d10c4197bd2c7543d9423970; no voters:
I20260619 02:49:32.090396 29046 leader_election.cc:288] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:32.090550 29056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:32.091991 29056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [term 1 LEADER]: Becoming Leader. State: Replica: 474368a7d10c4197bd2c7543d9423970, State: Running, Role: LEADER
I20260619 02:49:32.092473 29046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:32.092499 29056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } }
I20260619 02:49:32.095090 29058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 474368a7d10c4197bd2c7543d9423970. Latest consensus state: current_term: 1 leader_uuid: "474368a7d10c4197bd2c7543d9423970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } } }
I20260619 02:49:32.095400 29058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:32.095654 29057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "474368a7d10c4197bd2c7543d9423970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474368a7d10c4197bd2c7543d9423970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39105 } } }
I20260619 02:49:32.095959 29057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:32.098278 29063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:32.102545 29063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:32.111397 29063 catalog_manager.cc:1347] Generated new cluster ID: be4315c1bac04462add4c217cd091445
I20260619 02:49:32.111508 29063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:32.135438 29063 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:32.136682 29063 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:32.153810 29063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 474368a7d10c4197bd2c7543d9423970: Generated new TSK 0
I20260619 02:49:32.154636 29063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:32.166383 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28979
W20260619 02:49:32.167498 29041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43411
W20260619 02:49:32.168211 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 28979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:32.168352 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 28979
2026-06-19T02:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:32 dist-test-slave-9tkz krb5kdc[29077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:32 dist-test-slave-9tkz krb5kdc[29077](info): set up 2 sockets
Jun 19 02:49:32 dist-test-slave-9tkz krb5kdc[29077](info): commencing operation
krb5kdc: starting...
W20260619 02:49:33.270007 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.010s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:33 dist-test-slave-9tkz krb5kdc[29077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:33.516189 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:33.858342 29090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:33.858669 29090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:33.858839 29090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:33.866281 29090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:33.866428 29090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:33.866539 29090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:33.866657 29090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:33.866740 29090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:33.876569 29090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:33.877861 29090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:33.879136 29090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:33.891101 29098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.891130 29096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:33.891301 29095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:33.891829 29090 server_base.cc:1034] running on GCE node
I20260619 02:49:33.892621 29090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:33.894081 29090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:33.895300 29090 hybrid_clock.cc:648] HybridClock initialized: now 1781837373895237 us; error 48 us; skew 500 ppm
I20260619 02:49:33.895639 29090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:33 dist-test-slave-9tkz krb5kdc[29077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:33.902894 29090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:33.904332 29090 webserver.cc:460] Webserver started at http://127.0.0.1:39951/ using document root <none> and password file <none>
I20260619 02:49:33.904990 29090 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:33.905087 29090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:33.905336 29090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:33.908284 29090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "0d38b3f157a9430a9e614070d70c3e24"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-9tkz"
server_key: "be84c071645614868bff922ef137eed9"
server_key_iv: "04032787e0b192f32209fb608ac3cff4"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.908974 29090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "0d38b3f157a9430a9e614070d70c3e24"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-9tkz"
server_key: "be84c071645614868bff922ef137eed9"
server_key_iv: "04032787e0b192f32209fb608ac3cff4"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.914156 29090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:49:33.917639 29105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:33.918903 29090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:33.919072 29090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "0d38b3f157a9430a9e614070d70c3e24"
format_stamp: "Formatted at 2026-06-19 02:49:33 on dist-test-slave-9tkz"
server_key: "be84c071645614868bff922ef137eed9"
server_key_iv: "04032787e0b192f32209fb608ac3cff4"
server_key_version: "encryptionkey@0"
I20260619 02:49:33.919239 29090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:33.924822 29090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:33.928875 29090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:33.929155 29090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:33.946638 29090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42033
I20260619 02:49:33.946681 29156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42033 every 8 connection(s)
I20260619 02:49:33.947993 29090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:33.951620 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:33.951869 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29090
I20260619 02:49:33.952030 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:33.952464 8153 external_mini_cluster.cc:1428] Setting key 94aeea5b4e7c3eaca1d5b804db1dc4f3
I20260619 02:49:33.959883 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.954181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51049 (local address 127.0.0.1:42033)
0619 02:49:33.954988 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:33.954999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:33.955039 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:33.956540 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:33.956559 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:33.956891 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:33.957286 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:33.958370 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:33.958375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:33.958388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:33.958415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:33.958473 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:33.958494 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:33.958742 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:33.958876 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:49:33.959037 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":344,"threads_started":1}
I20260619 02:49:33.963605 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:33.965114 29157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24: Bootstrap starting.
I20260619 02:49:33.968436 29157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:33.969643 29157 log.cc:824] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:33.970425 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:33.967964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58959 (local address 127.0.0.1:42033)
0619 02:49:33.968133 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:33.968140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:33.968461 (+ 321us) server_negotiation.cc:407] Connection header received
0619 02:49:33.968656 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:33.968660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:33.968707 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:33.968819 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:33.969801 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:33.969813 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:33.969817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:33.969831 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:33.969870 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:33.969885 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:33.970004 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:33.970106 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:49:33.970166 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:33.972817 29157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24: No bootstrap required, opened a new log
I20260619 02:49:33.980243 29157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 02:49:33.980585 29157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:33.980656 29157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d38b3f157a9430a9e614070d70c3e24, State: Initialized, Role: FOLLOWER
I20260619 02:49:33.981153 29157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 02:49:33.981395 29157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:33.981570 29157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:33.981758 29157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:33.983996 29157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 02:49:33.984484 29157 leader_election.cc:302] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d38b3f157a9430a9e614070d70c3e24; no voters:
I20260619 02:49:33.985049 29157 leader_election.cc:288] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:33.985201 29167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:33.985935 29167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [term 1 LEADER]: Becoming Leader. State: Replica: 0d38b3f157a9430a9e614070d70c3e24, State: Running, Role: LEADER
I20260619 02:49:33.986449 29167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260619 02:49:33.987115 29157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:33.989334 29168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d38b3f157a9430a9e614070d70c3e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260619 02:49:33.989883 29168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:33.989764 29169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d38b3f157a9430a9e614070d70c3e24. Latest consensus state: current_term: 1 leader_uuid: "0d38b3f157a9430a9e614070d70c3e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d38b3f157a9430a9e614070d70c3e24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260619 02:49:33.990491 29169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:33.990738 29176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:33.997203 29176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:34.006556 29176 catalog_manager.cc:1347] Generated new cluster ID: 2205ddfe0eeb4092a77bb1f87c558ea4
I20260619 02:49:34.006685 29176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:34.024384 29176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:34.024842 29176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:34.032932 29176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d38b3f157a9430a9e614070d70c3e24: Generated new TSK 0
I20260619 02:49:34.033950 29176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:34.041433 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29090
W20260619 02:49:34.042758 29152 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51049
W20260619 02:49:34.043418 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:34.043531 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29090
2026-06-19T02:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:34 dist-test-slave-9tkz krb5kdc[29188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:34 dist-test-slave-9tkz krb5kdc[29188](info): set up 2 sockets
Jun 19 02:49:34 dist-test-slave-9tkz krb5kdc[29188](info): commencing operation
krb5kdc: starting...
W20260619 02:49:35.142112 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:35 dist-test-slave-9tkz krb5kdc[29188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:35.394927 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:35.724565 29201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:35.724937 29201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:35.725100 29201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:35.732326 29201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:35.732450 29201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:35.732497 29201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:35.732589 29201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:35.732681 29201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:35.742872 29201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:35.744172 29201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:35.745486 29201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:35.757249 29206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.757184 29207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:35.757347 29209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:35.758047 29201 server_base.cc:1034] running on GCE node
I20260619 02:49:35.758862 29201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:35.760252 29201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:35.761484 29201 hybrid_clock.cc:648] HybridClock initialized: now 1781837375761430 us; error 50 us; skew 500 ppm
I20260619 02:49:35.761842 29201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:35 dist-test-slave-9tkz krb5kdc[29188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:35.764894 29201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:35.766475 29201 webserver.cc:460] Webserver started at http://127.0.0.1:42151/ using document root <none> and password file <none>
I20260619 02:49:35.767160 29201 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:35.767256 29201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:35.767498 29201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:35.770450 29201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "fc1b352142244263a7446c8f0a9a1889"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-9tkz"
server_key: "662c04c29bbd92cd5e691474c8b83a66"
server_key_iv: "4a2fe85d7f064feb4013ea8f4fe02825"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.771169 29201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "fc1b352142244263a7446c8f0a9a1889"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-9tkz"
server_key: "662c04c29bbd92cd5e691474c8b83a66"
server_key_iv: "4a2fe85d7f064feb4013ea8f4fe02825"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.776419 29201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:35.780055 29216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:35.781579 29201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:35.781769 29201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "fc1b352142244263a7446c8f0a9a1889"
format_stamp: "Formatted at 2026-06-19 02:49:35 on dist-test-slave-9tkz"
server_key: "662c04c29bbd92cd5e691474c8b83a66"
server_key_iv: "4a2fe85d7f064feb4013ea8f4fe02825"
server_key_version: "encryptionkey@0"
I20260619 02:49:35.781955 29201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:35.815249 29201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:35.819624 29201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:35.819927 29201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:35.840358 29201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35975
I20260619 02:49:35.840387 29267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35975 every 8 connection(s)
I20260619 02:49:35.841826 29201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:35.845487 29268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:35.848735 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29201
I20260619 02:49:35.848913 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:35.849448 8153 external_mini_cluster.cc:1428] Setting key 4c062ee8b197b8e774433e5ee292104c
I20260619 02:49:35.857450 29268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889: Bootstrap starting.
I20260619 02:49:35.860657 29268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:35.861778 29268 log.cc:824] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:49:35 dist-test-slave-9tkz krb5kdc[29188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:49:35.864573 29268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889: No bootstrap required, opened a new log
I20260619 02:49:35.872090 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.851470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:45121 (local address 127.0.0.1:35975)
0619 02:49:35.852198 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:35.852218 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:35.852258 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:35.854043 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:35.854066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:35.854509 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:35.854924 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:35.856367 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.857755 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.859018 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.859728 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.860083 (+ 355us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.864276 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:35.864284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:35.864298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:35.864339 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:35.867500 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.868306 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.868313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.868318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.868497 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.868976 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.868980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.868985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.869504 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:35.869923 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:35.870398 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 02:49:35.870743 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
I20260619 02:49:35.873174 29268 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260619 02:49:35.873662 29268 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:35.873798 29268 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1b352142244263a7446c8f0a9a1889, State: Initialized, Role: FOLLOWER
I20260619 02:49:35.874367 29268 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260619 02:49:35.874626 29268 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:35.874787 29268 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:35.875008 29268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:35.876367 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:35.877887 29268 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260619 02:49:35.878460 29268 leader_election.cc:302] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1b352142244263a7446c8f0a9a1889; no voters:
I20260619 02:49:35.879112 29268 leader_election.cc:288] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:35.879212 29273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:35.881706 29268 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:35.882315 29273 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [term 1 LEADER]: Becoming Leader. State: Replica: fc1b352142244263a7446c8f0a9a1889, State: Running, Role: LEADER
I20260619 02:49:35.882823 29273 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } }
I20260619 02:49:35.886914 29276 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1b352142244263a7446c8f0a9a1889. Latest consensus state: current_term: 1 leader_uuid: "fc1b352142244263a7446c8f0a9a1889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } } }
I20260619 02:49:35.887293 29276 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:35.887859 29274 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1b352142244263a7446c8f0a9a1889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1b352142244263a7446c8f0a9a1889" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35975 } } }
I20260619 02:49:35.888139 29274 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:35.894742 29282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:35.900833 29282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:35.916356 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:35.895918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55475 (local address 127.0.0.1:35975)
0619 02:49:35.896113 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:35.896120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:35.897260 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 02:49:35.897545 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:35.897551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:35.897653 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:35.897733 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:35.899818 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.901002 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.902843 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:35.903471 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:35.903569 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:35.906956 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:35.906962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:35.906966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:35.907003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:35.911463 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.912752 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.912761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.912767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.912978 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:35.913727 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:35.913735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:35.913740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:35.914307 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:35.914635 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:35.915583 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 02:49:35.915806 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:49:35.922930 29282 catalog_manager.cc:1347] Generated new cluster ID: 830e17791b814767b53a3d4825d58406
I20260619 02:49:35.923046 29282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:35.937884 29282 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:35.939268 29282 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:35.953178 29282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1b352142244263a7446c8f0a9a1889: Generated new TSK 0
I20260619 02:49:35.954079 29282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:35.961217 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29201
I20260619 02:49:35.962327 29263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:36.052145 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29201
W20260619 02:49:36.069703 29186 connection.cc:535] client connection to 127.0.0.1:35975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35975): Connection reset by peer (error 104)
2026-06-19T02:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:36 dist-test-slave-9tkz krb5kdc[29300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:36 dist-test-slave-9tkz krb5kdc[29300](info): set up 2 sockets
Jun 19 02:49:36 dist-test-slave-9tkz krb5kdc[29300](info): commencing operation
krb5kdc: starting...
W20260619 02:49:37.158043 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:37 dist-test-slave-9tkz krb5kdc[29300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:37.412585 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:37.738804 29313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.739177 29313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.739349 29313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.746677 29313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:37.746842 29313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.746949 29313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.747061 29313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:37.747148 29313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:37.756938 29313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:37.758248 29313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.759570 29313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.771124 29318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.771095 29319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.771050 29321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.772174 29313 server_base.cc:1034] running on GCE node
I20260619 02:49:37.772998 29313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.774504 29313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.775712 29313 hybrid_clock.cc:648] HybridClock initialized: now 1781837377775666 us; error 44 us; skew 500 ppm
I20260619 02:49:37.776067 29313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:37 dist-test-slave-9tkz krb5kdc[29300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:37.779192 29313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:37.780561 29313 webserver.cc:460] Webserver started at http://127.0.0.1:37789/ using document root <none> and password file <none>
I20260619 02:49:37.781210 29313 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.781304 29313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.781620 29313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.784595 29313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "b6212ba85d214c8b958e83be46c765ab"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-9tkz"
server_key: "62825886a3593970559a0f98b95fba5d"
server_key_iv: "47225377808d097a1038b6fededd6646"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.785312 29313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "b6212ba85d214c8b958e83be46c765ab"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-9tkz"
server_key: "62825886a3593970559a0f98b95fba5d"
server_key_iv: "47225377808d097a1038b6fededd6646"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.790416 29313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:49:37.793584 29328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.794898 29313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:49:37.795063 29313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "b6212ba85d214c8b958e83be46c765ab"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-9tkz"
server_key: "62825886a3593970559a0f98b95fba5d"
server_key_iv: "47225377808d097a1038b6fededd6646"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.795218 29313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.810595 29313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.814628 29313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.814951 29313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.834111 29313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36089
I20260619 02:49:37.834136 29379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36089 every 8 connection(s)
I20260619 02:49:37.835891 29313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:37.837160 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29313
I20260619 02:49:37.837325 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:37.837932 8153 external_mini_cluster.cc:1428] Setting key 48a872ac8973135a7fb025b293759077
I20260619 02:49:37.840509 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.850714 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.839761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:58397 (local address 127.0.0.1:36089)
0619 02:49:37.840482 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.840494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.840539 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:49:37.842247 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.842266 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.842631 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.843025 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.843563 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.844860 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.846234 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.846905 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.847108 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.848461 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.848469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.848482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.849045 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.849114 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.849147 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.849337 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.849636 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.849799 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260619 02:49:37.851420 29380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab: Bootstrap starting.
I20260619 02:49:37.854656 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:37.854876 29380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.855923 29380 log.cc:824] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.859414 29380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab: No bootstrap required, opened a new log
I20260619 02:49:37.865178 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.859022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:41593 (local address 127.0.0.1:36089)
0619 02:49:37.859175 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.859179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.859666 (+ 487us) server_negotiation.cc:407] Connection header received
0619 02:49:37.859892 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.859896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.859958 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.860064 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.860565 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.861344 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.862738 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.863196 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.863340 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.864450 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.864455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.864460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:37.864477 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.864529 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:37.864544 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.864670 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.864854 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.864907 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:49:37.867846 29380 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } }
I20260619 02:49:37.868338 29380 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.868477 29380 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6212ba85d214c8b958e83be46c765ab, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.869135 29380 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } }
I20260619 02:49:37.869391 29380 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:37.869535 29380 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:37.869769 29380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.871889 29380 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } }
I20260619 02:49:37.872388 29380 leader_election.cc:302] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6212ba85d214c8b958e83be46c765ab; no voters:
I20260619 02:49:37.872985 29380 leader_election.cc:288] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:37.873077 29390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:37.873790 29390 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [term 1 LEADER]: Becoming Leader. State: Replica: b6212ba85d214c8b958e83be46c765ab, State: Running, Role: LEADER
I20260619 02:49:37.874363 29390 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } }
I20260619 02:49:37.875330 29380 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:37.877285 29392 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6212ba85d214c8b958e83be46c765ab. Latest consensus state: current_term: 1 leader_uuid: "b6212ba85d214c8b958e83be46c765ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } } }
I20260619 02:49:37.877676 29392 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.877383 29391 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6212ba85d214c8b958e83be46c765ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6212ba85d214c8b958e83be46c765ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36089 } } }
I20260619 02:49:37.877839 29391 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.880296 29395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:37.885901 29395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:37.894996 29395 catalog_manager.cc:1347] Generated new cluster ID: e701ff21114f4ca590790fa5178cd8e3
I20260619 02:49:37.895147 29395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:37.932403 29395 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:37.933781 29395 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:37.952129 29395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6212ba85d214c8b958e83be46c765ab: Generated new TSK 0
I20260619 02:49:37.953086 29395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:37.958774 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29313
W20260619 02:49:37.960275 29375 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:58397
W20260619 02:49:37.961094 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29313: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:37.961263 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29313
W20260619 02:49:37.978641 29298 connection.cc:535] client connection to 127.0.0.1:36089 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36089): Connection reset by peer (error 104)
2026-06-19T02:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:38 dist-test-slave-9tkz krb5kdc[29411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:38 dist-test-slave-9tkz krb5kdc[29411](info): set up 2 sockets
Jun 19 02:49:38 dist-test-slave-9tkz krb5kdc[29411](info): commencing operation
krb5kdc: starting...
W20260619 02:49:39.070001 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:39 dist-test-slave-9tkz krb5kdc[29411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:39.329365 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:39.661083 29424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:39.661399 29424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:39.661585 29424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:39.668970 29424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:39.669082 29424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:39.669184 29424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:39.669268 29424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:39.669349 29424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:39.679369 29424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:39.680682 29424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:39.682022 29424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:39.693670 29429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.693614 29430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.693694 29432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.694794 29424 server_base.cc:1034] running on GCE node
I20260619 02:49:39.695544 29424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:39.696923 29424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:39.698185 29424 hybrid_clock.cc:648] HybridClock initialized: now 1781837379698125 us; error 45 us; skew 500 ppm
I20260619 02:49:39.698778 29424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:39 dist-test-slave-9tkz krb5kdc[29411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:39.708050 29424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:39.709832 29424 webserver.cc:460] Webserver started at http://127.0.0.1:33449/ using document root <none> and password file <none>
I20260619 02:49:39.710541 29424 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:39.710695 29424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:39.711010 29424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:39.714112 29424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "503158bbb38740aea8b79bf869a7cf00"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-9tkz"
server_key: "3ac192648d25546916c023b35696d504"
server_key_iv: "cc2d78645ff3f22817db0dbc0bf28746"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.714854 29424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "503158bbb38740aea8b79bf869a7cf00"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-9tkz"
server_key: "3ac192648d25546916c023b35696d504"
server_key_iv: "cc2d78645ff3f22817db0dbc0bf28746"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.720098 29424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:49:39.723523 29439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.724797 29424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:39.724967 29424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "503158bbb38740aea8b79bf869a7cf00"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-9tkz"
server_key: "3ac192648d25546916c023b35696d504"
server_key_iv: "cc2d78645ff3f22817db0dbc0bf28746"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.725154 29424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:39.730947 29424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:39.735316 29424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:39.735605 29424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:39.754053 29424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37809
I20260619 02:49:39.754077 29490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37809 every 8 connection(s)
I20260619 02:49:39.755512 29424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:39.759109 29491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:39.764132 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29424
I20260619 02:49:39.764356 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:39.764880 8153 external_mini_cluster.cc:1428] Setting key 10ebb84ea70f7e433cea09997cbcff2e
I20260619 02:49:39.770429 29491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00: Bootstrap starting.
I20260619 02:49:39.772847 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.766770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:43969 (local address 127.0.0.1:37809)
0619 02:49:39.767579 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.767590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.767631 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:39.769381 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.769402 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.769841 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.770222 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.771257 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.771263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.771275 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:39.771287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.771343 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:39.771360 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.771581 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.771760 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.771961 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":354,"threads_started":1}
I20260619 02:49:39.773761 29491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:39.774833 29491 log.cc:824] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:39.776630 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:39.778373 29491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00: No bootstrap required, opened a new log
I20260619 02:49:39.784475 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.782006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:51097 (local address 127.0.0.1:37809)
0619 02:49:39.782195 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.782202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.782522 (+ 320us) server_negotiation.cc:407] Connection header received
0619 02:49:39.782666 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.782676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.782759 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.782879 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.783771 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.783777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.783782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:39.783801 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.783860 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:39.783880 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.784023 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.784071 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.784129 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:49:39.788038 29491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260619 02:49:39.788422 29491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:39.788544 29491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503158bbb38740aea8b79bf869a7cf00, State: Initialized, Role: FOLLOWER
I20260619 02:49:39.789189 29491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260619 02:49:39.789438 29491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:39.789619 29491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:39.789808 29491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:39.792132 29491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260619 02:49:39.792625 29491 leader_election.cc:302] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503158bbb38740aea8b79bf869a7cf00; no voters:
I20260619 02:49:39.793196 29491 leader_election.cc:288] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:39.793318 29501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:39.794039 29501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [term 1 LEADER]: Becoming Leader. State: Replica: 503158bbb38740aea8b79bf869a7cf00, State: Running, Role: LEADER
I20260619 02:49:39.794593 29501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260619 02:49:39.795279 29491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:39.797843 29502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "503158bbb38740aea8b79bf869a7cf00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } } }
I20260619 02:49:39.798202 29502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:39.801779 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503158bbb38740aea8b79bf869a7cf00. Latest consensus state: current_term: 1 leader_uuid: "503158bbb38740aea8b79bf869a7cf00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503158bbb38740aea8b79bf869a7cf00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } } }
I20260619 02:49:39.802129 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:39.803284 29507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:39.808395 29507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:39.817512 29507 catalog_manager.cc:1347] Generated new cluster ID: fa0e36b8003a4d4aa95566b0a5cb14ba
I20260619 02:49:39.817703 29507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:39.827859 29507 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:39.828317 29507 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:39.837154 29507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503158bbb38740aea8b79bf869a7cf00: Generated new TSK 0
I20260619 02:49:39.838157 29507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:39.852058 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29424
W20260619 02:49:39.853133 29486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:43969
W20260619 02:49:39.853935 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29424: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:39.854076 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29424
2026-06-19T02:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:39 dist-test-slave-9tkz krb5kdc[29522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:39 dist-test-slave-9tkz krb5kdc[29522](info): set up 2 sockets
Jun 19 02:49:39 dist-test-slave-9tkz krb5kdc[29522](info): commencing operation
krb5kdc: starting...
W20260619 02:49:40.986117 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:41.229104 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:41.551147 29535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:41.551465 29535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:41.551635 29535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:41.558954 29535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:41.559103 29535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:41.559202 29535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:41.559314 29535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:41.559399 29535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:41.569197 29535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:41.570544 29535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:41.571913 29535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:41.584012 29540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.583979 29541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.583930 29543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.584946 29535 server_base.cc:1034] running on GCE node
I20260619 02:49:41.585868 29535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:41.587342 29535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:41.588567 29535 hybrid_clock.cc:648] HybridClock initialized: now 1781837381588529 us; error 32 us; skew 500 ppm
I20260619 02:49:41.588917 29535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.592000 29535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:41.593844 29535 webserver.cc:460] Webserver started at http://127.0.0.1:36545/ using document root <none> and password file <none>
I20260619 02:49:41.594550 29535 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:41.594663 29535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:41.594931 29535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:41.598116 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "e12bf87a6be74240bee7d2b79f329ddf"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-9tkz"
server_key: "59b4e06ebe8eac8c3269d6a60625c69d"
server_key_iv: "251ed529a74dbcfd4626c5636567067e"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.598904 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "e12bf87a6be74240bee7d2b79f329ddf"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-9tkz"
server_key: "59b4e06ebe8eac8c3269d6a60625c69d"
server_key_iv: "251ed529a74dbcfd4626c5636567067e"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.604269 29535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:49:41.607731 29550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.609017 29535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:41.609170 29535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "e12bf87a6be74240bee7d2b79f329ddf"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-9tkz"
server_key: "59b4e06ebe8eac8c3269d6a60625c69d"
server_key_iv: "251ed529a74dbcfd4626c5636567067e"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.609350 29535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:41.627378 29535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:41.631229 29535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:41.631458 29535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:41.650468 29535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36575
I20260619 02:49:41.650492 29601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36575 every 8 connection(s)
I20260619 02:49:41.651849 29535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:41.652777 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29535
I20260619 02:49:41.652951 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:41.653339 8153 external_mini_cluster.cc:1428] Setting key 739eca4494a486a61843fc8c2c0fecb7
I20260619 02:49:41.656361 29602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:49:41.670192 29602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf: Bootstrap starting.
I20260619 02:49:41.674372 29602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.675761 29602 log.cc:824] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:41.677788 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.655133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:57961 (local address 127.0.0.1:36575)
0619 02:49:41.655940 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.655952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.656000 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:41.657757 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.657778 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.658157 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.658490 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.660279 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.661704 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.663514 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.664135 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.664253 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.669116 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.669127 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.669148 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.669197 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.673135 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.673979 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.673989 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.673998 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.674230 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.674723 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.674730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.674735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.675450 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.675766 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.676145 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.676564 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260619 02:49:41.679164 29602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf: No bootstrap required, opened a new log
I20260619 02:49:41.681641 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:41.687566 29602 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260619 02:49:41.687940 29602 raft_consensus.cc:374] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.688045 29602 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12bf87a6be74240bee7d2b79f329ddf, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.688772 29602 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260619 02:49:41.689041 29602 raft_consensus.cc:388] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:41.689181 29602 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:41.689348 29602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.692083 29602 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260619 02:49:41.692626 29602 leader_election.cc:302] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e12bf87a6be74240bee7d2b79f329ddf; no voters:
I20260619 02:49:41.693220 29602 leader_election.cc:288] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:41.693372 29612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:41.695545 29602 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:41.700281 29612 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [term 1 LEADER]: Becoming Leader. State: Replica: e12bf87a6be74240bee7d2b79f329ddf, State: Running, Role: LEADER
I20260619 02:49:41.700995 29612 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260619 02:49:41.703094 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.685871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:54067 (local address 127.0.0.1:36575)
0619 02:49:41.686072 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.686079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.686529 (+ 450us) server_negotiation.cc:407] Connection header received
0619 02:49:41.686665 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.686670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.686754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.686851 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.688286 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.689475 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.690711 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.691376 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.691493 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.694683 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.694690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.694695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.694731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.699002 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.700072 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.700080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.700090 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.700298 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.700870 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.700875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.700877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.701306 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.701673 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.702295 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.702529 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:49:41.704273 29613 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e12bf87a6be74240bee7d2b79f329ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260619 02:49:41.704622 29613 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:41.707218 29625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:41.707075 29614 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12bf87a6be74240bee7d2b79f329ddf. Latest consensus state: current_term: 1 leader_uuid: "e12bf87a6be74240bee7d2b79f329ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e12bf87a6be74240bee7d2b79f329ddf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260619 02:49:41.707383 29614 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:41.711584 29625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:41.719848 29625 catalog_manager.cc:1347] Generated new cluster ID: 31c190902ae24d40899649a5d12a9753
I20260619 02:49:41.719966 29625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:41.736238 29625 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:41.737509 29625 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:41.748392 29625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12bf87a6be74240bee7d2b79f329ddf: Generated new TSK 0
I20260619 02:49:41.749181 29625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:41.757294 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29535
I20260619 02:49:41.758585 29597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:41.848448 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29535
W20260619 02:49:41.866235 29520 connection.cc:535] client connection to 127.0.0.1:36575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36575): Connection reset by peer (error 104)
2026-06-19T02:49:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29634](info): set up 2 sockets
Jun 19 02:49:41 dist-test-slave-9tkz krb5kdc[29634](info): commencing operation
krb5kdc: starting...
W20260619 02:49:42.970487 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:43 dist-test-slave-9tkz krb5kdc[29634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:43.224952 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.554368 29647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.554651 29647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.554765 29647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.561873 29647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:43.561985 29647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.562032 29647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.562093 29647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:43.562135 29647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:43.571779 29647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:43.573019 29647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.574226 29647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.586048 29653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.586048 29655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.586732 29647 server_base.cc:1034] running on GCE node
W20260619 02:49:43.586823 29652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.587704 29647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.589020 29647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.590238 29647 hybrid_clock.cc:648] HybridClock initialized: now 1781837383590200 us; error 56 us; skew 500 ppm
I20260619 02:49:43.590593 29647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:43 dist-test-slave-9tkz krb5kdc[29634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.593701 29647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:43.595149 29647 webserver.cc:460] Webserver started at http://127.0.0.1:33871/ using document root <none> and password file <none>
I20260619 02:49:43.595845 29647 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.595991 29647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.596294 29647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.599303 29647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "461c5cce40054e2cba6db353042c992a"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-9tkz"
server_key: "bab2ec4f146b7f365a5108c2d5dc4f9b"
server_key_iv: "bc892efe123a6f1663951e22ef635103"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.600104 29647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "461c5cce40054e2cba6db353042c992a"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-9tkz"
server_key: "bab2ec4f146b7f365a5108c2d5dc4f9b"
server_key_iv: "bc892efe123a6f1663951e22ef635103"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.605470 29647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:43.609285 29662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.610705 29647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:43.610905 29647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "461c5cce40054e2cba6db353042c992a"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-9tkz"
server_key: "bab2ec4f146b7f365a5108c2d5dc4f9b"
server_key_iv: "bc892efe123a6f1663951e22ef635103"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.611109 29647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.624603 29647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.628541 29647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.628808 29647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.647287 29647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33223
I20260619 02:49:43.647365 29713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33223 every 8 connection(s)
I20260619 02:49:43.648630 29647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:43.652391 29714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.658785 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29647
I20260619 02:49:43.658975 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:43.659381 8153 external_mini_cluster.cc:1428] Setting key 9098c6653e41551c707b22e8fff665b1
I20260619 02:49:43.663682 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: Bootstrap starting.
I20260619 02:49:43.667445 29714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.668967 29714 log.cc:824] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.672400 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: No bootstrap required, opened a new log
I20260619 02:49:43.673700 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.661170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:51035 (local address 127.0.0.1:33223)
0619 02:49:43.662001 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.662012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.662051 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:49:43.663652 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.663677 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.664095 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.664408 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.665032 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.666371 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.668379 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.669314 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.669442 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.670890 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.670901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.670920 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:43.671632 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.671751 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:43.671788 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.671970 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.672218 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.672378 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":403,"threads_started":1}
I20260619 02:49:43.677747 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:43.683475 29714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260619 02:49:43.683969 29714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.684104 29714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461c5cce40054e2cba6db353042c992a, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.684813 29714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260619 02:49:43.685137 29714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:43.685316 29714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:43.685518 29714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:43.688460 29714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260619 02:49:43.689061 29714 leader_election.cc:302] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461c5cce40054e2cba6db353042c992a; no voters:
I20260619 02:49:43.689939 29714 leader_election.cc:288] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:43.690003 29724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:43.690104 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.682858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:49685 (local address 127.0.0.1:33223)
0619 02:49:43.683059 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.683067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.683584 (+ 517us) server_negotiation.cc:407] Connection header received
0619 02:49:43.683790 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.683794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.683848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.683988 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.684518 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.685814 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.687078 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.687867 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.688003 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.689095 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.689102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.689106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:43.689129 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.689175 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:43.689198 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.689323 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.689537 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.689655 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:49:43.692271 29714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.696950 29724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [term 1 LEADER]: Becoming Leader. State: Replica: 461c5cce40054e2cba6db353042c992a, State: Running, Role: LEADER
I20260619 02:49:43.697470 29724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } }
I20260619 02:49:43.700774 29725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461c5cce40054e2cba6db353042c992a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260619 02:49:43.701328 29725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.700783 29726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461c5cce40054e2cba6db353042c992a. Latest consensus state: current_term: 1 leader_uuid: "461c5cce40054e2cba6db353042c992a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461c5cce40054e2cba6db353042c992a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33223 } } }
I20260619 02:49:43.701766 29726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:43.710280 29738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:43.710399 29738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:43.710531 29740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:43.712354 29740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:43.720988 29740 catalog_manager.cc:1347] Generated new cluster ID: 4022e7f53d1748e78e7f7a5537c4e030
I20260619 02:49:43.721120 29740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:43.751350 29740 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:43.752662 29740 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:43.762861 29740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461c5cce40054e2cba6db353042c992a: Generated new TSK 0
I20260619 02:49:43.763675 29740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:43.771806 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29647
W20260619 02:49:43.772784 29709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:51035
W20260619 02:49:43.773494 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29647: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:43.773685 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29647
W20260619 02:49:43.790385 29632 connection.cc:535] client connection to 127.0.0.1:33223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33223): Connection reset by peer (error 104)
2026-06-19T02:49:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:43 dist-test-slave-9tkz krb5kdc[29745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:43 dist-test-slave-9tkz krb5kdc[29745](info): set up 2 sockets
Jun 19 02:49:43 dist-test-slave-9tkz krb5kdc[29745](info): commencing operation
krb5kdc: starting...
W20260619 02:49:44.882046 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:45 dist-test-slave-9tkz krb5kdc[29745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:49:45.132162 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:45.460269 29758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:45.460563 29758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:45.460657 29758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:45.467965 29758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:45.468086 29758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:45.468135 29758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:45.468190 29758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:45.468235 29758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:45.478197 29758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:45.479487 29758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:45.480819 29758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:45.492496 29764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:45.492542 29766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:45.492513 29763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:45.493300 29758 server_base.cc:1034] running on GCE node
I20260619 02:49:45.494200 29758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:45.495611 29758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:45.496869 29758 hybrid_clock.cc:648] HybridClock initialized: now 1781837385496814 us; error 38 us; skew 500 ppm
I20260619 02:49:45.497305 29758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:45 dist-test-slave-9tkz krb5kdc[29745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:45.504945 29758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:49:45.506709 29758 webserver.cc:460] Webserver started at http://127.0.0.1:40945/ using document root <none> and password file <none>
I20260619 02:49:45.507480 29758 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:45.507643 29758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:45.508004 29758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:45.511269 29758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "7c2027cabdb442b1843aaee8b12dee2d"
format_stamp: "Formatted at 2026-06-19 02:49:45 on dist-test-slave-9tkz"
server_key: "dcf68817f696e50dcaa900fd69b23c1d"
server_key_iv: "9c8b7911e4639d5975974ed0ce587616"
server_key_version: "encryptionkey@0"
I20260619 02:49:45.512115 29758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "7c2027cabdb442b1843aaee8b12dee2d"
format_stamp: "Formatted at 2026-06-19 02:49:45 on dist-test-slave-9tkz"
server_key: "dcf68817f696e50dcaa900fd69b23c1d"
server_key_iv: "9c8b7911e4639d5975974ed0ce587616"
server_key_version: "encryptionkey@0"
I20260619 02:49:45.517774 29758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:49:45.521441 29773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:45.522856 29758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:45.523051 29758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "7c2027cabdb442b1843aaee8b12dee2d"
format_stamp: "Formatted at 2026-06-19 02:49:45 on dist-test-slave-9tkz"
server_key: "dcf68817f696e50dcaa900fd69b23c1d"
server_key_iv: "9c8b7911e4639d5975974ed0ce587616"
server_key_version: "encryptionkey@0"
I20260619 02:49:45.523259 29758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:45.529052 29758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:45.532932 29758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:45.533226 29758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:45.551841 29758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44935
I20260619 02:49:45.551882 29824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44935 every 8 connection(s)
I20260619 02:49:45.553264 29758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:45.555827 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29758
I20260619 02:49:45.556016 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:45.556535 8153 external_mini_cluster.cc:1428] Setting key f6dca23ddcbccf27e0832ad743981637
I20260619 02:49:45.557374 29825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:45.564127 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.558247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:60909 (local address 127.0.0.1:44935)
0619 02:49:45.558991 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.559003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.559043 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:49:45.560587 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.560608 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.560948 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.561375 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:45.562463 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:45.562469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:45.562481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:45.562495 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:45.562574 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:45.562593 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:45.562850 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.562996 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.563187 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":299,"threads_started":1}
I20260619 02:49:45.567932 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:45.570001 29825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: Bootstrap starting.
I20260619 02:49:45.573426 29825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:45.574821 29825 log.cc:824] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:45.575999 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:45.573059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55827 (local address 127.0.0.1:44935)
0619 02:49:45.573280 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:45.573289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:45.573712 (+ 423us) server_negotiation.cc:407] Connection header received
0619 02:49:45.573892 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:45.573899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:45.573969 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:45.574082 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:45.575163 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:45.575169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:45.575173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:45.575193 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:45.575240 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:45.575266 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:45.575410 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:45.575621 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:49:45.575692 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:49:45.578114 29825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: No bootstrap required, opened a new log
I20260619 02:49:45.588198 29825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } }
I20260619 02:49:45.588699 29825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:45.588814 29825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2027cabdb442b1843aaee8b12dee2d, State: Initialized, Role: FOLLOWER
I20260619 02:49:45.589579 29825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } }
I20260619 02:49:45.589814 29825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:45.589958 29825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:45.590142 29825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:45.593722 29825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } }
I20260619 02:49:45.594420 29825 leader_election.cc:302] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c2027cabdb442b1843aaee8b12dee2d; no voters:
I20260619 02:49:45.595147 29825 leader_election.cc:288] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:45.595264 29835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:45.596114 29835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [term 1 LEADER]: Becoming Leader. State: Replica: 7c2027cabdb442b1843aaee8b12dee2d, State: Running, Role: LEADER
I20260619 02:49:45.596777 29835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } }
I20260619 02:49:45.597579 29825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:45.600382 29837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c2027cabdb442b1843aaee8b12dee2d. Latest consensus state: current_term: 1 leader_uuid: "7c2027cabdb442b1843aaee8b12dee2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } } }
I20260619 02:49:45.600747 29837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:45.601502 29836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c2027cabdb442b1843aaee8b12dee2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c2027cabdb442b1843aaee8b12dee2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44935 } } }
I20260619 02:49:45.601930 29836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:45.614739 29851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:45.614909 29851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:45.615814 29841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:45.618165 29841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:45.627113 29841 catalog_manager.cc:1347] Generated new cluster ID: 3045efbe49974380866ca46bb3a59f3c
I20260619 02:49:45.627235 29841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:45.651135 29841 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:45.651578 29841 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:45.664671 29841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c2027cabdb442b1843aaee8b12dee2d: Generated new TSK 0
I20260619 02:49:45.665513 29841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:45.676909 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29758
W20260619 02:49:45.678020 29820 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.198:60909
W20260619 02:49:45.678684 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29758: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:45.678830 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29758
2026-06-19T02:49:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:45 dist-test-slave-9tkz krb5kdc[29856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:45 dist-test-slave-9tkz krb5kdc[29856](info): set up 2 sockets
Jun 19 02:49:45 dist-test-slave-9tkz krb5kdc[29856](info): commencing operation
krb5kdc: starting...
W20260619 02:49:46.781999 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:46 dist-test-slave-9tkz krb5kdc[29856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:47.028805 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45837
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:47.353468 29869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:47.353885 29869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:47.354061 29869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:47.361179 29869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:47.361315 29869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:47.361418 29869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:47.361531 29869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:47.361622 29869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:47.371541 29869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:45837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:47.372802 29869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:47.374106 29869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:47.386423 29874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.386466 29877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:47.386512 29875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:47.393435 29869 server_base.cc:1034] running on GCE node
I20260619 02:49:47.394301 29869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:47.395754 29869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:47.396965 29869 hybrid_clock.cc:648] HybridClock initialized: now 1781837387396923 us; error 30 us; skew 500 ppm
I20260619 02:49:47.397363 29869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:47 dist-test-slave-9tkz krb5kdc[29856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837387, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:47.400489 29869 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:47.402143 29869 webserver.cc:460] Webserver started at http://127.7.246.126:46487/ using document root <none> and password file <none>
I20260619 02:49:47.402858 29869 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:47.403005 29869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:47.403316 29869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:47.406303 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-9tkz"
server_key: "98958f955abe8a1b65084e32f8cc66a9"
server_key_iv: "0cc6db0029d04fc074ce93d7a79ed4fc"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.407047 29869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-9tkz"
server_key: "98958f955abe8a1b65084e32f8cc66a9"
server_key_iv: "0cc6db0029d04fc074ce93d7a79ed4fc"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.412016 29869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:49:47.415388 29884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:47.416668 29869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:49:47.416860 29869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c"
format_stamp: "Formatted at 2026-06-19 02:49:47 on dist-test-slave-9tkz"
server_key: "98958f955abe8a1b65084e32f8cc66a9"
server_key_iv: "0cc6db0029d04fc074ce93d7a79ed4fc"
server_key_version: "encryptionkey@0"
I20260619 02:49:47.417058 29869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:47.437098 29869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:47.440912 29869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:47.441187 29869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:47.460824 29869 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:45837
I20260619 02:49:47.460826 29935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:45837 every 8 connection(s)
I20260619 02:49:47.462230 29869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:47.463127 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29869
I20260619 02:49:47.463318 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:47.463855 8153 external_mini_cluster.cc:1428] Setting key b2bfa5bf7094a0314f226418d2e64c83
I20260619 02:49:47.466620 29936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:49:47 dist-test-slave-9tkz krb5kdc[29856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:47.479586 29936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c: Bootstrap starting.
I20260619 02:49:47.483706 29936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:47.484926 29936 log.cc:824] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:47.488179 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.465598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51671 (local address 127.7.246.126:45837)
0619 02:49:47.466407 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.466419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.466458 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:49:47.468083 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.468103 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.468476 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.468789 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.470507 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.471846 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.473614 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.474288 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.474414 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.479300 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.479312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.479333 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.479387 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.483322 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.484263 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.484271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.484278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.484483 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.484963 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.484968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.484971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.485693 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.485930 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.486600 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.486914 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":356,"threads_started":1}
I20260619 02:49:47.488333 29936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c: No bootstrap required, opened a new log
I20260619 02:49:47.492455 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:47.496153 29936 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } }
I20260619 02:49:47.496510 29936 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:47.496623 29936 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dbfa13955f4c4c8b0d3773cdee9c1c, State: Initialized, Role: FOLLOWER
I20260619 02:49:47.497274 29936 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } }
I20260619 02:49:47.497514 29936 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:47.497727 29936 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:47.497937 29936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:47.500591 29936 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } }
I20260619 02:49:47.501152 29936 leader_election.cc:302] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4dbfa13955f4c4c8b0d3773cdee9c1c; no voters:
I20260619 02:49:47.501809 29936 leader_election.cc:288] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:47.501909 29946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:47.505494 29946 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [term 1 LEADER]: Becoming Leader. State: Replica: e4dbfa13955f4c4c8b0d3773cdee9c1c, State: Running, Role: LEADER
I20260619 02:49:47.506029 29936 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:47.506172 29946 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } }
I20260619 02:49:47.509505 29947 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } } }
I20260619 02:49:47.509637 29948 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4dbfa13955f4c4c8b0d3773cdee9c1c. Latest consensus state: current_term: 1 leader_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4dbfa13955f4c4c8b0d3773cdee9c1c" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45837 } } }
I20260619 02:49:47.509989 29948 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:47.509989 29947 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:47.514544 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:47.496920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37265 (local address 127.7.246.126:45837)
0619 02:49:47.497153 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:47.497158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:47.497608 (+ 450us) server_negotiation.cc:407] Connection header received
0619 02:49:47.497829 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:47.497835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:47.497921 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:47.498027 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:47.499485 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.500588 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.502047 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:47.502765 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:47.502886 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:47.505427 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:47.505435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:47.505442 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:47.505499 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:47.509253 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.510299 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.510304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.510307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.510504 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:47.511276 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:47.511284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:47.511290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:47.511782 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:47.513441 (+ 1659us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:47.513681 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 02:49:47.513925 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 02:49:47.521683 29952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:47.529064 29952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:47.537452 29952 catalog_manager.cc:1347] Generated new cluster ID: aadea33a98b54cf5b88a38d33b6b3409
I20260619 02:49:47.537611 29952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:47.555521 29952 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:47.556881 29952 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:47.565765 29952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4dbfa13955f4c4c8b0d3773cdee9c1c: Generated new TSK 0
I20260619 02:49:47.566836 29952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:47.583339 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29869
I20260619 02:49:47.585078 29931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:47.676867 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29869
W20260619 02:49:47.693676 29854 connection.cc:535] client connection to 127.7.246.126:45837 recv error: Network error: failed to read from TLS socket (remote: 127.7.246.126:45837): Connection reset by peer (error 104)
2026-06-19T02:49:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:47 dist-test-slave-9tkz krb5kdc[29968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:47 dist-test-slave-9tkz krb5kdc[29968](info): set up 2 sockets
Jun 19 02:49:47 dist-test-slave-9tkz krb5kdc[29968](info): commencing operation
krb5kdc: starting...
W20260619 02:49:48.786079 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:48 dist-test-slave-9tkz krb5kdc[29968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:49.034678 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45407
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:45407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:49.353466 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:49.353787 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:49.353891 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:49.361172 29981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:49.361279 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:49.361323 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:49.361382 29981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:49.361419 29981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:49.372252 29981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:45407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:45407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:49.373504 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:49.374792 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:49.386713 29986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.386765 29989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.386796 29987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.387650 29981 server_base.cc:1034] running on GCE node
I20260619 02:49:49.388330 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:49.389652 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:49.390862 29981 hybrid_clock.cc:648] HybridClock initialized: now 1781837389390815 us; error 40 us; skew 500 ppm
I20260619 02:49:49.391207 29981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:49 dist-test-slave-9tkz krb5kdc[29968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:49.394212 29981 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:49.395655 29981 webserver.cc:460] Webserver started at http://127.7.246.126:41477/ using document root <none> and password file <none>
I20260619 02:49:49.396373 29981 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:49.396512 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:49.396803 29981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:49.399722 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "5f0a41a68eb9419c8a717e0993157bc2"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-9tkz"
server_key: "84b0ed8dd3f68e8adb59c18f63e212ad"
server_key_iv: "8a9b74f0e7698929c6377800e6a3c47c"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.400632 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "5f0a41a68eb9419c8a717e0993157bc2"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-9tkz"
server_key: "84b0ed8dd3f68e8adb59c18f63e212ad"
server_key_iv: "8a9b74f0e7698929c6377800e6a3c47c"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.405673 29981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:49:49.408994 29996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.410256 29981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:49.410424 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "5f0a41a68eb9419c8a717e0993157bc2"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-9tkz"
server_key: "84b0ed8dd3f68e8adb59c18f63e212ad"
server_key_iv: "8a9b74f0e7698929c6377800e6a3c47c"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.410626 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:49.431646 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:49.435465 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:49.435753 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:49.454092 29981 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:45407
I20260619 02:49:49.454130 30047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:45407 every 8 connection(s)
I20260619 02:49:49.455734 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:49.457363 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 29981
I20260619 02:49:49.457580 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:49.458133 8153 external_mini_cluster.cc:1428] Setting key ae9ac7a7f9dca4a0f173eba549c83887
I20260619 02:49:49.460498 30048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:49.470721 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.459975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33635 (local address 127.7.246.126:45407)
0619 02:49:49.460791 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.460802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.460843 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:49.462500 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.462520 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.462890 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.463288 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.463804 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.465127 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.466503 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.467172 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.467366 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.468566 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.468573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.468585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:49.469098 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.469172 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:49.469198 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.469380 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.469674 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.469832 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":337,"threads_started":1}
I20260619 02:49:49.471410 30048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2: Bootstrap starting.
I20260619 02:49:49.474309 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:49.474754 30048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:49.475705 30048 log.cc:824] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:49.478880 30048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2: No bootstrap required, opened a new log
I20260619 02:49:49.484853 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.478770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55299 (local address 127.7.246.126:45407)
0619 02:49:49.478920 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.478925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.479405 (+ 480us) server_negotiation.cc:407] Connection header received
0619 02:49:49.479609 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.479614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.479669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.479776 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.480250 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.481059 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.482407 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.482874 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.483034 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.484151 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.484156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.484159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:49.484175 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.484209 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:49.484226 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.484359 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.484536 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.484590 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:49:49.487326 30048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } }
I20260619 02:49:49.487706 30048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:49.487836 30048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0a41a68eb9419c8a717e0993157bc2, State: Initialized, Role: FOLLOWER
I20260619 02:49:49.488399 30048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } }
I20260619 02:49:49.488618 30048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:49.488768 30048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:49.488951 30048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:49.491554 30048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } }
I20260619 02:49:49.492085 30048 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0a41a68eb9419c8a717e0993157bc2; no voters:
I20260619 02:49:49.492643 30048 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:49.492766 30058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:49.493562 30058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 5f0a41a68eb9419c8a717e0993157bc2, State: Running, Role: LEADER
I20260619 02:49:49.494139 30058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } }
I20260619 02:49:49.494896 30048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:49.498080 30059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0a41a68eb9419c8a717e0993157bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } } }
I20260619 02:49:49.498458 30059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:49.499161 30060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0a41a68eb9419c8a717e0993157bc2. Latest consensus state: current_term: 1 leader_uuid: "5f0a41a68eb9419c8a717e0993157bc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0a41a68eb9419c8a717e0993157bc2" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 45407 } } }
I20260619 02:49:49.499473 30060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:49.502058 30068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:49.507586 30068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:49.516374 30068 catalog_manager.cc:1347] Generated new cluster ID: 25fd0171b399456aa296a272bf788bbf
I20260619 02:49:49.516505 30068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:49.528846 30068 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:49.530229 30068 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:49.539173 30068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0a41a68eb9419c8a717e0993157bc2: Generated new TSK 0
I20260619 02:49:49.540251 30068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:49.555369 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29981
W20260619 02:49:49.556989 30043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33635
W20260619 02:49:49.557806 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 29981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:49.557979 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 29981
W20260619 02:49:49.575238 29966 connection.cc:535] client connection to 127.7.246.126:45407 recv error: Network error: failed to read from TLS socket (remote: 127.7.246.126:45407): Connection reset by peer (error 104)
2026-06-19T02:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:49 dist-test-slave-9tkz krb5kdc[30079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:49 dist-test-slave-9tkz krb5kdc[30079](info): set up 2 sockets
Jun 19 02:49:49 dist-test-slave-9tkz krb5kdc[30079](info): commencing operation
krb5kdc: starting...
W20260619 02:49:50.670122 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:50 dist-test-slave-9tkz krb5kdc[30079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:50.937533 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43275
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:51.262979 30092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:51.263362 30092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:51.263549 30092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:51.271317 30092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:51.271518 30092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:51.271647 30092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:51.271768 30092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:51.271854 30092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:51.282557 30092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:51.284024 30092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:51.285432 30092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:51.298278 30100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:51.298324 30098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:51.298324 30097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:51.299819 30092 server_base.cc:1034] running on GCE node
I20260619 02:49:51.300607 30092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:51.302101 30092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:51.303359 30092 hybrid_clock.cc:648] HybridClock initialized: now 1781837391303303 us; error 43 us; skew 500 ppm
I20260619 02:49:51.303797 30092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:51 dist-test-slave-9tkz krb5kdc[30079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837391, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:51.307268 30092 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:51.308758 30092 webserver.cc:460] Webserver started at http://127.7.246.126:43069/ using document root <none> and password file <none>
I20260619 02:49:51.309504 30092 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:51.309669 30092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:51.309958 30092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:51.312826 30092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "a84ce80042774340bb30dce761a6634b"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-9tkz"
server_key: "01682dd71bed0ad76820830ff6d66269"
server_key_iv: "f900566da4c67adba6649ec89adc94e9"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.313593 30092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "a84ce80042774340bb30dce761a6634b"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-9tkz"
server_key: "01682dd71bed0ad76820830ff6d66269"
server_key_iv: "f900566da4c67adba6649ec89adc94e9"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.318665 30092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 02:49:51.322213 30107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:51.323580 30092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:51.323808 30092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "a84ce80042774340bb30dce761a6634b"
format_stamp: "Formatted at 2026-06-19 02:49:51 on dist-test-slave-9tkz"
server_key: "01682dd71bed0ad76820830ff6d66269"
server_key_iv: "f900566da4c67adba6649ec89adc94e9"
server_key_version: "encryptionkey@0"
I20260619 02:49:51.324028 30092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:51.329733 30092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:51.334060 30092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:51.334365 30092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:51.352514 30092 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43275
I20260619 02:49:51.352543 30158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43275 every 8 connection(s)
I20260619 02:49:51.354028 30092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:51.357810 30159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:51.360471 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30092
I20260619 02:49:51.360641 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:51.361127 8153 external_mini_cluster.cc:1428] Setting key 2b4207fd31c720fd420aa925dcfc4843
I20260619 02:49:51.368963 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.363001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43507 (local address 127.7.246.126:43275)
0619 02:49:51.363846 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.363858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.363904 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:49:51.365523 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.365599 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.365954 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.366304 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.367383 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.367390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.367402 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:51.367420 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.367488 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:51.367510 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.367700 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.367812 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.368034 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":400,"threads_started":1}
I20260619 02:49:51.370334 30159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b: Bootstrap starting.
I20260619 02:49:51.372773 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:51.373664 30159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:51.374641 30159 log.cc:824] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:51.378115 30159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b: No bootstrap required, opened a new log
I20260619 02:49:51.379951 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.377247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46553 (local address 127.7.246.126:43275)
0619 02:49:51.377441 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.377449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.377907 (+ 458us) server_negotiation.cc:407] Connection header received
0619 02:49:51.378109 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.378118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.378187 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.378303 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.379280 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.379286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.379290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:51.379307 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.379354 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:51.379370 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.379500 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.379612 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.379679 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:49:51.386212 30159 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } }
I20260619 02:49:51.386579 30159 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:51.386690 30159 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84ce80042774340bb30dce761a6634b, State: Initialized, Role: FOLLOWER
I20260619 02:49:51.387339 30159 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } }
I20260619 02:49:51.387585 30159 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:51.387729 30159 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:51.387879 30159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:51.390015 30159 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } }
I20260619 02:49:51.390437 30159 leader_election.cc:302] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84ce80042774340bb30dce761a6634b; no voters:
I20260619 02:49:51.390949 30159 leader_election.cc:288] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:51.391108 30169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:51.391804 30169 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [term 1 LEADER]: Becoming Leader. State: Replica: a84ce80042774340bb30dce761a6634b, State: Running, Role: LEADER
I20260619 02:49:51.392323 30169 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } }
I20260619 02:49:51.392999 30159 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:51.395804 30170 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84ce80042774340bb30dce761a6634b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } } }
I20260619 02:49:51.395381 30171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84ce80042774340bb30dce761a6634b. Latest consensus state: current_term: 1 leader_uuid: "a84ce80042774340bb30dce761a6634b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84ce80042774340bb30dce761a6634b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43275 } } }
I20260619 02:49:51.396525 30171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:51.396525 30170 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:51.409519 30177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:51.414466 30177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:51.423074 30177 catalog_manager.cc:1347] Generated new cluster ID: 0b768660c8584850bf7f29aeb4845ba8
I20260619 02:49:51.423198 30177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:51.450382 30177 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:51.450917 30177 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:51.461760 30177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84ce80042774340bb30dce761a6634b: Generated new TSK 0
I20260619 02:49:51.462833 30177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:51.472802 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30092
W20260619 02:49:51.473888 30154 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43507
W20260619 02:49:51.474570 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30092: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:51.474725 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30092
2026-06-19T02:49:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:51 dist-test-slave-9tkz krb5kdc[30190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:51 dist-test-slave-9tkz krb5kdc[30190](info): set up 2 sockets
Jun 19 02:49:51 dist-test-slave-9tkz krb5kdc[30190](info): commencing operation
krb5kdc: starting...
W20260619 02:49:52.578184 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:52 dist-test-slave-9tkz krb5kdc[30190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:52.832684 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38899
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:38899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:53.158015 30203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:53.158289 30203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:53.158486 30203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:53.165769 30203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:53.165874 30203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:53.165999 30203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:53.166136 30203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:53.166222 30203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:53.175872 30203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:38899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:38899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:53.177196 30203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:53.178550 30203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:53.190621 30208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:53.190696 30209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:53.190639 30211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:53.191565 30203 server_base.cc:1034] running on GCE node
I20260619 02:49:53.192528 30203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:53.193989 30203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:53.195226 30203 hybrid_clock.cc:648] HybridClock initialized: now 1781837393195198 us; error 47 us; skew 500 ppm
I20260619 02:49:53.195575 30203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:53 dist-test-slave-9tkz krb5kdc[30190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837393, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:53.198704 30203 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:53.200182 30203 webserver.cc:460] Webserver started at http://127.7.246.126:40207/ using document root <none> and password file <none>
I20260619 02:49:53.200851 30203 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:53.201004 30203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:53.201305 30203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:53.204290 30203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "b9334123f491442ab5cd8ec5a04d38f3"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-9tkz"
server_key: "6675b2279bdc8161260a02807debebd9"
server_key_iv: "8f784b47f1a2d9299b33d01e06205576"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.205054 30203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "b9334123f491442ab5cd8ec5a04d38f3"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-9tkz"
server_key: "6675b2279bdc8161260a02807debebd9"
server_key_iv: "8f784b47f1a2d9299b33d01e06205576"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.210235 30203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:49:53.213680 30218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:53.214901 30203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:53.215088 30203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "b9334123f491442ab5cd8ec5a04d38f3"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-9tkz"
server_key: "6675b2279bdc8161260a02807debebd9"
server_key_iv: "8f784b47f1a2d9299b33d01e06205576"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.215286 30203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:53.227913 30203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:53.231921 30203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:53.232208 30203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:53.250804 30203 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:38899
I20260619 02:49:53.250824 30269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:38899 every 8 connection(s)
I20260619 02:49:53.252193 30203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:53.255911 30270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:53.257911 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30203
I20260619 02:49:53.258078 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:53.258553 8153 external_mini_cluster.cc:1428] Setting key 4c5f980db1f6ab4b0c2028aa57c1c1f3
I20260619 02:49:53.268246 30270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3: Bootstrap starting.
I20260619 02:49:53.272254 30270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3: Neither blocks nor log segments found. Creating new log.
Jun 19 02:49:53 dist-test-slave-9tkz krb5kdc[30190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:49:53.273427 30270 log.cc:824] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:53.276283 30270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3: No bootstrap required, opened a new log
I20260619 02:49:53.281814 30273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.260402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50595 (local address 127.7.246.126:38899)
0619 02:49:53.261213 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.261223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.261266 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:49:53.262973 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.262995 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.263372 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.263675 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.265454 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.266831 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.268638 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.269283 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.269413 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.274227 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:53.274234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:53.274248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:53.274282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:53.277893 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.278544 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.278551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.278557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.278719 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.279118 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.279123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.279132 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.279643 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:53.279978 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:53.280369 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 02:49:53.280784 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":405,"threads_started":1}
I20260619 02:49:53.285049 30270 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } }
I20260619 02:49:53.285465 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:53.285451 30270 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:53.285590 30270 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9334123f491442ab5cd8ec5a04d38f3, State: Initialized, Role: FOLLOWER
I20260619 02:49:53.286183 30270 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } }
I20260619 02:49:53.286414 30270 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:53.286563 30270 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:53.286731 30270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:53.289048 30270 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } }
I20260619 02:49:53.289525 30270 leader_election.cc:302] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9334123f491442ab5cd8ec5a04d38f3; no voters:
I20260619 02:49:53.290231 30270 leader_election.cc:288] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:53.290390 30279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:53.292229 30270 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:53.295008 30279 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [term 1 LEADER]: Becoming Leader. State: Replica: b9334123f491442ab5cd8ec5a04d38f3, State: Running, Role: LEADER
I20260619 02:49:53.295655 30279 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } }
I20260619 02:49:53.298386 30281 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9334123f491442ab5cd8ec5a04d38f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } } }
I20260619 02:49:53.298766 30281 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:53.305042 30288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:53.305680 30282 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9334123f491442ab5cd8ec5a04d38f3. Latest consensus state: current_term: 1 leader_uuid: "b9334123f491442ab5cd8ec5a04d38f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9334123f491442ab5cd8ec5a04d38f3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 38899 } } }
I20260619 02:49:53.306017 30282 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:53.312779 30288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:53.314669 30273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.290132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39063 (local address 127.7.246.126:38899)
0619 02:49:53.290330 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.290337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.294505 (+ 4168us) server_negotiation.cc:407] Connection header received
0619 02:49:53.294819 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.294827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.294933 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.295048 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.297039 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.298701 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.301265 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.302256 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.302412 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.306203 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:53.306211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:53.306215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:53.306263 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:53.310724 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.311482 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.311487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.311491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.311647 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.312064 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.312069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.312072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.312486 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:53.313922 (+ 1436us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:53.314066 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 02:49:53.314300 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:49:53.323405 30288 catalog_manager.cc:1347] Generated new cluster ID: de917d2fdc254dbfb5f3a98881da3d25
I20260619 02:49:53.323524 30288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:53.379312 30288 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:53.380609 30288 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:53.387450 30288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9334123f491442ab5cd8ec5a04d38f3: Generated new TSK 0
I20260619 02:49:53.388288 30288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:53.394335 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30203
I20260619 02:49:53.395395 30265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:49:53.490243 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30203
2026-06-19T02:49:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:53 dist-test-slave-9tkz krb5kdc[30303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:53 dist-test-slave-9tkz krb5kdc[30303](info): set up 2 sockets
Jun 19 02:49:53 dist-test-slave-9tkz krb5kdc[30303](info): commencing operation
krb5kdc: starting...
W20260619 02:49:54.602157 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:54 dist-test-slave-9tkz krb5kdc[30303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:54.861253 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36379
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:55.188743 30316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:55.189041 30316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:55.189211 30316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:55.196544 30316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:55.196650 30316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:55.196754 30316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:55.196863 30316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:55.196944 30316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:55.206923 30316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:55.208252 30316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:55.210045 30316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:55.221638 30321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:55.221670 30322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:55.221881 30324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:55.223104 30316 server_base.cc:1034] running on GCE node
I20260619 02:49:55.223984 30316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:55.225713 30316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:55.226974 30316 hybrid_clock.cc:648] HybridClock initialized: now 1781837395226935 us; error 50 us; skew 500 ppm
I20260619 02:49:55.227366 30316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:55 dist-test-slave-9tkz krb5kdc[30303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:55.230964 30316 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:55.232496 30316 webserver.cc:460] Webserver started at http://127.7.246.126:36317/ using document root <none> and password file <none>
I20260619 02:49:55.233189 30316 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:55.233300 30316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:55.233644 30316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:55.236984 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "d18c3433ed404a329f224ac1a67cfcc0"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-9tkz"
server_key: "6c318fa3379bccf08fe51ddd274426cd"
server_key_iv: "be12734f096d1280b64fc4a4cce7f7c8"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.237883 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "d18c3433ed404a329f224ac1a67cfcc0"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-9tkz"
server_key: "6c318fa3379bccf08fe51ddd274426cd"
server_key_iv: "be12734f096d1280b64fc4a4cce7f7c8"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.243424 30316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:49:55.247238 30331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:55.248664 30316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:55.248896 30316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "d18c3433ed404a329f224ac1a67cfcc0"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-9tkz"
server_key: "6c318fa3379bccf08fe51ddd274426cd"
server_key_iv: "be12734f096d1280b64fc4a4cce7f7c8"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.249111 30316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:55.276625 30316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:55.281286 30316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:55.281639 30316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:55.302424 30316 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:36379
I20260619 02:49:55.302430 30382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:36379 every 8 connection(s)
I20260619 02:49:55.303960 30316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:55.306619 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30316
I20260619 02:49:55.306809 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:55.307329 8153 external_mini_cluster.cc:1428] Setting key 461ba5891db1e6daa5cf37f70d6e0ce7
I20260619 02:49:55.307901 30383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:55.322486 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0: Bootstrap starting.
I20260619 02:49:55.322688 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.309272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43627 (local address 127.7.246.126:36379)
0619 02:49:55.310102 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.310117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.310165 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:55.312522 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.312549 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.313068 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.313496 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.314037 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.316056 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.317455 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.318534 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.318724 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.319945 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:55.319959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:55.319984 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:55.320701 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:55.320793 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:55.320822 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:55.320947 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:55.321147 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:49:55.321390 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":371,"threads_started":1}
I20260619 02:49:55.325762 30383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:55.326743 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:55.327072 30383 log.cc:824] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:55.330447 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0: No bootstrap required, opened a new log
I20260619 02:49:55.338826 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.331766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32797 (local address 127.7.246.126:36379)
0619 02:49:55.331981 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.331989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.332331 (+ 342us) server_negotiation.cc:407] Connection header received
0619 02:49:55.332561 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.332565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.332632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.332738 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.333330 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.334211 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.336145 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.336672 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.336776 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.337948 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:55.337953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:55.337956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:55.337972 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:55.338008 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:55.338023 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:55.338093 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:55.338265 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 02:49:55.338338 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:49:55.341002 30383 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } }
I20260619 02:49:55.341364 30383 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:55.341437 30383 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18c3433ed404a329f224ac1a67cfcc0, State: Initialized, Role: FOLLOWER
I20260619 02:49:55.342196 30383 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } }
I20260619 02:49:55.342490 30383 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:55.342633 30383 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:55.342803 30383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:55.344969 30383 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } }
I20260619 02:49:55.345424 30383 leader_election.cc:302] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18c3433ed404a329f224ac1a67cfcc0; no voters:
I20260619 02:49:55.346082 30383 leader_election.cc:288] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:55.346225 30393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:55.347215 30393 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [term 1 LEADER]: Becoming Leader. State: Replica: d18c3433ed404a329f224ac1a67cfcc0, State: Running, Role: LEADER
I20260619 02:49:55.347724 30393 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } }
I20260619 02:49:55.348300 30383 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:55.350455 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d18c3433ed404a329f224ac1a67cfcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } } }
I20260619 02:49:55.350916 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:55.351320 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18c3433ed404a329f224ac1a67cfcc0. Latest consensus state: current_term: 1 leader_uuid: "d18c3433ed404a329f224ac1a67cfcc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c3433ed404a329f224ac1a67cfcc0" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 36379 } } }
I20260619 02:49:55.351601 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:55.351761 30402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:55.359138 30402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:55.370617 30402 catalog_manager.cc:1347] Generated new cluster ID: 0f65c9df5c194a7d93b68536033c4ad1
I20260619 02:49:55.370735 30402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:55.405048 30402 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:55.406456 30402 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:55.413144 30402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d18c3433ed404a329f224ac1a67cfcc0: Generated new TSK 0
I20260619 02:49:55.413996 30402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:55.419970 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30316
W20260619 02:49:55.421032 30378 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43627
W20260619 02:49:55.421756 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30316: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:55.421901 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30316
2026-06-19T02:49:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:55 dist-test-slave-9tkz krb5kdc[30414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:55 dist-test-slave-9tkz krb5kdc[30414](info): set up 2 sockets
Jun 19 02:49:55 dist-test-slave-9tkz krb5kdc[30414](info): commencing operation
krb5kdc: starting...
W20260619 02:49:56.534484 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:56 dist-test-slave-9tkz krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:56.788383 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40259
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:40259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:57.120803 30427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:57.121189 30427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:57.121372 30427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:57.128849 30427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:57.129025 30427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:57.129135 30427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:57.129245 30427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:57.129338 30427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:57.139556 30427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:40259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:57.140962 30427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:57.142375 30427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:57.154662 30432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.154631 30433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.154728 30435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.155645 30427 server_base.cc:1034] running on GCE node
I20260619 02:49:57.156533 30427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:57.158082 30427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:57.159381 30427 hybrid_clock.cc:648] HybridClock initialized: now 1781837397159317 us; error 51 us; skew 500 ppm
I20260619 02:49:57.159795 30427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:57 dist-test-slave-9tkz krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837397, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:57.168567 30427 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:57.170358 30427 webserver.cc:460] Webserver started at http://127.7.246.126:39601/ using document root <none> and password file <none>
I20260619 02:49:57.171154 30427 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:57.171319 30427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:57.171643 30427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:57.174889 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "66788dfedd8642f99b565d9ec634659b"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-9tkz"
server_key: "aff6001efd2f3f262dfb35a5789f85b9"
server_key_iv: "c8ba56cad1606d2746ac49ff7b625ffa"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.175684 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "66788dfedd8642f99b565d9ec634659b"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-9tkz"
server_key: "aff6001efd2f3f262dfb35a5789f85b9"
server_key_iv: "c8ba56cad1606d2746ac49ff7b625ffa"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.181107 30427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:49:57.185292 30442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:57.186734 30427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:57.186946 30427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "66788dfedd8642f99b565d9ec634659b"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-9tkz"
server_key: "aff6001efd2f3f262dfb35a5789f85b9"
server_key_iv: "c8ba56cad1606d2746ac49ff7b625ffa"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.187157 30427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:57.193480 30427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:57.198082 30427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:57.198340 30427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:57.217188 30427 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:40259
I20260619 02:49:57.217221 30493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:40259 every 8 connection(s)
I20260619 02:49:57.218734 30427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:57.222422 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30427
I20260619 02:49:57.222548 30494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:57.222599 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:57.223017 8153 external_mini_cluster.cc:1428] Setting key 85dc2a34d705150c07d11f8f52b5af93
I20260619 02:49:57.231410 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.224978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54139 (local address 127.7.246.126:40259)
0619 02:49:57.225820 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.225834 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.225903 (+ 69us) server_negotiation.cc:407] Connection header received
0619 02:49:57.227594 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.227621 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.228077 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.228506 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.229483 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.229492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.229512 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.229528 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.229616 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.229659 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.229892 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.230033 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.230218 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":335,"threads_started":1}
I20260619 02:49:57.235072 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:57.237115 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b: Bootstrap starting.
I20260619 02:49:57.240643 30494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:57.241870 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.239389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49083 (local address 127.7.246.126:40259)
0619 02:49:57.239555 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.239561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.239916 (+ 355us) server_negotiation.cc:407] Connection header received
0619 02:49:57.240044 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.240049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.240107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.240206 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.241120 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.241126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.241130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.241147 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.241191 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.241209 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.241322 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.241483 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.241539 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:49:57.241966 30494 log.cc:824] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:57.244638 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b: No bootstrap required, opened a new log
I20260619 02:49:57.251725 30494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } }
I20260619 02:49:57.252061 30494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:57.252130 30494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66788dfedd8642f99b565d9ec634659b, State: Initialized, Role: FOLLOWER
I20260619 02:49:57.252604 30494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } }
I20260619 02:49:57.252794 30494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:57.252878 30494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:57.252993 30494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.255385 30494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } }
I20260619 02:49:57.256006 30494 leader_election.cc:302] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66788dfedd8642f99b565d9ec634659b; no voters:
I20260619 02:49:57.256712 30494 leader_election.cc:288] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:57.256840 30504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.264120 30494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:57.265092 30504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [term 1 LEADER]: Becoming Leader. State: Replica: 66788dfedd8642f99b565d9ec634659b, State: Running, Role: LEADER
I20260619 02:49:57.267156 30504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } }
I20260619 02:49:57.271025 30506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66788dfedd8642f99b565d9ec634659b. Latest consensus state: current_term: 1 leader_uuid: "66788dfedd8642f99b565d9ec634659b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } } }
I20260619 02:49:57.271608 30506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.272225 30505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66788dfedd8642f99b565d9ec634659b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66788dfedd8642f99b565d9ec634659b" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 40259 } } }
I20260619 02:49:57.272521 30505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.274924 30513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:57.281730 30513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:57.304210 30513 catalog_manager.cc:1347] Generated new cluster ID: f1689cc307b24aa7ad07993832f95eb5
I20260619 02:49:57.304339 30513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:57.322314 30513 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:57.322834 30513 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:57.335862 30513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66788dfedd8642f99b565d9ec634659b: Generated new TSK 0
I20260619 02:49:57.336987 30513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:57.348369 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30427
W20260619 02:49:57.349488 30489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54139
W20260619 02:49:57.350313 8153 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30427: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:49:57.350476 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30427
2026-06-19T02:49:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1928 ms)
[----------] 18 tests from AuthTokenIssuingTest (34830 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:57 dist-test-slave-9tkz krb5kdc[30526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:57 dist-test-slave-9tkz krb5kdc[30526](info): set up 2 sockets
Jun 19 02:49:57 dist-test-slave-9tkz krb5kdc[30526](info): commencing operation
krb5kdc: starting...
W20260619 02:49:58.454384 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:49:58 dist-test-slave-9tkz krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:49:58.717279 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43021
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:59.045799 30539 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:49:59.046092 30539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.046245 30539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.046394 30539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:59.053491 30539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:59.053642 30539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.053781 30539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.053877 30539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:59.053968 30539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:59.063616 30539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:59.064935 30539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.066262 30539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.078083 30547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.078045 30544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.078208 30545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.079779 30539 server_base.cc:1034] running on GCE node
I20260619 02:49:59.080489 30539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.081842 30539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.083109 30539 hybrid_clock.cc:648] HybridClock initialized: now 1781837399083050 us; error 36 us; skew 500 ppm
I20260619 02:49:59.083441 30539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.086588 30539 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:49:59.088043 30539 webserver.cc:460] Webserver started at http://127.7.246.126:43747/ using document root <none> and password file <none>
I20260619 02:49:59.088732 30539 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.088860 30539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.089167 30539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.092067 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "886eedf066c2428d87708b4c90b93b70"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "5e33050f65a6a02adc6f181e1211f3a5"
server_key_iv: "2b1a1a0a97e2935ebd557d17cc332928"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.092890 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "886eedf066c2428d87708b4c90b93b70"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "5e33050f65a6a02adc6f181e1211f3a5"
server_key_iv: "2b1a1a0a97e2935ebd557d17cc332928"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.098021 30539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:49:59.101470 30554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.103040 30539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:59.103259 30539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "886eedf066c2428d87708b4c90b93b70"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "5e33050f65a6a02adc6f181e1211f3a5"
server_key_iv: "2b1a1a0a97e2935ebd557d17cc332928"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.103502 30539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.116171 30539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.120555 30539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.120841 30539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.139780 30539 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:43021
I20260619 02:49:59.139799 30605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:43021 every 8 connection(s)
I20260619 02:49:59.141134 30539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:59.141853 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30539
I20260619 02:49:59.142084 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:49:59.142817 8153 external_mini_cluster.cc:1428] Setting key 74192f254f8c8a00f6453234383bd98f
I20260619 02:49:59.145087 30606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:59.151587 30606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.125@KRBTEST.COM" created.
I20260619 02:49:59.174731 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.154764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43021 (local address 127.0.0.1:47074)
0619 02:49:59.155618 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.155662 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.157739 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.158576 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.158583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.159293 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.159607 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.159624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.161163 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.161168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.162302 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.162314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.163209 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.163215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.163385 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.164097 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.164120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.166629 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.170561 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.170572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.170582 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.170982 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.171488 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.171491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.171493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.171646 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.172664 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.172670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.172861 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.173078 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.173496 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":369,"threads_started":1}
I20260619 02:49:59.174765 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.154764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.7.246.126:43021)
0619 02:49:59.156412 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.156420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.156469 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:49:59.158103 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.158108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.158340 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.158567 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.159828 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.161003 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.162441 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.162975 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.163170 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.166884 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.166893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.166905 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.166936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.170357 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.171162 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.171174 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.171180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.171370 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.171788 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.171796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.171802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.172526 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.172743 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.173308 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.173734 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":1422,"thread_start_us":1229,"threads_started":1}
I20260619 02:49:59.177907 30606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:49:59.179111 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.178591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38242)
0619 02:49:59.178817 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.178961 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 02:49:59.179389 30555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
I20260619 02:49:59.181021 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.180642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38248)
0619 02:49:59.180817 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.180903 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 02:49:59.181243 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:43175: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
I20260619 02:49:59.203059 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:43175
--webserver_interface=127.7.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:49:59.230597 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } attempt: 1
I20260619 02:49:59.231798 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.231322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38252)
0619 02:49:59.231515 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.231627 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:49:59.232877 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.232508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38260)
0619 02:49:59.232652 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.232751 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:49:59.233141 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:43175: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
I20260619 02:49:59.301414 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } attempt: 2
I20260619 02:49:59.302894 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.302367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38276)
0619 02:49:59.302633 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.302758 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 02:49:59.303974 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.303597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38278)
0619 02:49:59.303753 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.303840 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 02:49:59.304215 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:43175: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
I20260619 02:49:59.395498 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } attempt: 3
I20260619 02:49:59.396927 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.396384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38292)
0619 02:49:59.396631 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.396760 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 02:49:59.398144 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.397747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38300)
0619 02:49:59.397923 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.397997 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 02:49:59.398413 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:43175: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
W20260619 02:49:59.534370 30613 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:49:59.534746 30613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.534888 30613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.535073 30613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:49:59.541782 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } attempt: 4
I20260619 02:49:59.543160 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.542570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38310)
0619 02:49:59.542895 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.543010 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260619 02:49:59.543452 30613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:59.543587 30613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.543692 30613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.543817 30613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:59.543928 30613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:59.544373 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.543981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38326)
0619 02:49:59.544158 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.544249 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 02:49:59.544585 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:43175: Network error: Client connection negotiation failed: client connection to 127.7.246.125:43175: connect: Connection refused (error 111)
I20260619 02:49:59.555116 30613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:49:59.556506 30613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.558017 30613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.570053 30621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.570025 30619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.570151 30618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.570891 30613 server_base.cc:1034] running on GCE node
I20260619 02:49:59.571672 30613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.573086 30613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.574369 30613 hybrid_clock.cc:648] HybridClock initialized: now 1781837399574321 us; error 59 us; skew 500 ppm
I20260619 02:49:59.574733 30613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.577965 30613 init.cc:377] Logged in from keytab as kudu/127.7.246.125@KRBTEST.COM (short username kudu)
I20260619 02:49:59.579506 30613 webserver.cc:460] Webserver started at http://127.7.246.125:38155/ using document root <none> and password file <none>
I20260619 02:49:59.580186 30613 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.580307 30613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.580644 30613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.583741 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data/instance:
uuid: "444681890f254c7a87d5ae24b22d5490"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "016aa44c62b13e934e5bef9521453601"
server_key_iv: "3eae88649f4d526e9f51b8c6c8f8881a"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.584620 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal/instance:
uuid: "444681890f254c7a87d5ae24b22d5490"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "016aa44c62b13e934e5bef9521453601"
server_key_iv: "3eae88649f4d526e9f51b8c6c8f8881a"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.589707 30613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:49:59.593029 30628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.594475 30613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:59.594648 30613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "444681890f254c7a87d5ae24b22d5490"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-9tkz"
server_key: "016aa44c62b13e934e5bef9521453601"
server_key_iv: "3eae88649f4d526e9f51b8c6c8f8881a"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.594841 30613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.611668 30613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.615981 30613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.616355 30613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.635181 30613 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.125:43175
I20260619 02:49:59.635212 30679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.125:43175 every 8 connection(s)
I20260619 02:49:59.636550 30613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
I20260619 02:49:59.638800 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30613
I20260619 02:49:59.638980 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-1/wal/instance
I20260619 02:49:59.639396 8153 external_mini_cluster.cc:1428] Setting key 2b408e66489b14b96471c5bf0b6f1c2b
I20260619 02:49:59.640467 30680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:59.647037 30680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.124@KRBTEST.COM" created.
I20260619 02:49:59.669543 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.650379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.7.246.126:43021)
0619 02:49:59.650627 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.650633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.651709 (+ 1076us) server_negotiation.cc:407] Connection header received
0619 02:49:59.653616 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.653625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.653720 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.653843 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.655321 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.656433 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.658028 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.658876 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.659019 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.662546 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.662556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.662564 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.662612 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.665858 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.666827 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.666831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.666834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.666952 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.667570 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.667574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.667577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.667925 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.668069 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.669067 (+ 998us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.669280 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 02:49:59.671198 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.650747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43021 (local address 127.0.0.1:47088)
0619 02:49:59.651522 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.651571 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.653301 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.654032 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.654039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.654597 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.655134 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.655150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.656589 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.656593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.657864 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.657876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.659111 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.659116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.659309 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.660048 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.660070 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.662300 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.665993 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.666005 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.666026 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.666672 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.667106 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.667114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.667121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.667403 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.668103 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.668114 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.668604 (+ 490us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.668924 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.669544 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260619 02:49:59.673120 30680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:49:59.689182 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.673843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38328)
0619 02:49:59.674073 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.674097 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.674217 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.675400 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.675403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.675901 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.676124 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.676138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.677563 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.677571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.679158 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.679175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.680209 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.680217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.680463 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.681074 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.681101 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.683615 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.686821 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.686829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.686834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.687195 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.687596 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.687603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.687606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.687728 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.688319 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.688328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.688490 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.688647 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.688925 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:49:59.689723 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.674121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.7.246.125:43175)
0619 02:49:59.674851 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.674862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.674895 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:59.674951 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.674962 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.675219 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.675391 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.676318 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.677395 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.679324 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.680045 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.680139 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.683837 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.683850 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.683864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.683891 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.686681 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.687336 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.687371 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.687374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.687483 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.687834 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.687841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.687844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.688193 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.688440 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.688870 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.689155 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":373,"threads_started":1}
I20260619 02:49:59.692171 30680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:49:59.693419 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.692996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:34972)
0619 02:49:59.693168 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.693267 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:49:59.693737 30632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.695302 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.694938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:34988)
0619 02:49:59.695083 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.695165 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:49:59.695524 30680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
I20260619 02:49:59.702702 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:33853
--webserver_interface=127.7.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:49:59.744854 30680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 1
I20260619 02:49:59.746021 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.745532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35004)
0619 02:49:59.745766 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.745876 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 02:49:59.746989 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.746657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35008)
0619 02:49:59.746801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.746891 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:49:59.747184 30680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.815414 30680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 2
I20260619 02:49:59.816505 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.816055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35024)
0619 02:49:59.816254 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.816353 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 02:49:59.817474 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.817148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35038)
0619 02:49:59.817293 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.817367 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:49:59.817750 30680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.843868 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } attempt: 5
Jun 19 02:49:59 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:49:59.858264 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.844709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38336)
0619 02:49:59.844966 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.844995 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.845102 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.845455 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.845458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.845908 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.846096 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.846110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.847507 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.847510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.848373 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.848384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.849357 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.849361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.849524 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.850230 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.850248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.852231 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.855929 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.855932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.855935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.856273 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.856664 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.856667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.856669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.856788 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.857451 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.857455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.857679 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.857819 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.858014 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:49:59.858700 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.844892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.7.246.125:43175)
0619 02:49:59.845084 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.845090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.845116 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:49:59.845203 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.845210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.845308 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.845424 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.846273 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.847363 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.848524 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.849201 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.849325 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.852452 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.852458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.852463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.852509 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.855783 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.856394 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.856402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.856406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.856555 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.856891 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.856896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.856899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.857282 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.857426 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.857964 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.858250 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:49:59.859933 30606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:49:59.860997 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.860608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35042)
0619 02:49:59.860782 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.860869 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:49:59.861974 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.861664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35054)
0619 02:49:59.861797 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.861872 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 02:49:59.862182 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.909009 30680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 3
I20260619 02:49:59.910247 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.909757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35056)
0619 02:49:59.909998 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.910093 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 02:49:59.911429 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.911009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35068)
0619 02:49:59.911241 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.911318 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260619 02:49:59.911702 30680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.913422 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 1
I20260619 02:49:59.914417 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.914002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35082)
0619 02:49:59.914173 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.914261 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:49:59.915526 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.915151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35090)
0619 02:49:59.915313 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.915401 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:49:59.915776 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:49:59.984055 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 2
I20260619 02:49:59.985299 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.984836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35102)
0619 02:49:59.985049 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.985147 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:49:59.986310 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.985977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35116)
0619 02:49:59.986124 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.986211 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:49:59.986569 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
W20260619 02:50:00.038522 30687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:50:00.038875 30687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:00.038991 30687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:00.039120 30687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:00.046376 30687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:00.046463 30687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:00.046578 30687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:00.046665 30687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:00.046770 30687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:00.054929 30680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 4
I20260619 02:50:00.056041 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.055584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35118)
0619 02:50:00.055809 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.055901 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:50:00.057039 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.056740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35134)
0619 02:50:00.056874 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.056943 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:50:00.057085 30687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:43021,127.7.246.125:43175,127.7.246.124:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
W20260619 02:50:00.057332 30680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:50:00.058498 30687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:00.059803 30687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:00.071768 30695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:00.071836 30693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:00.071743 30692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:00.079056 30687 server_base.cc:1034] running on GCE node
I20260619 02:50:00.079864 30687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:00.081410 30687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:00.082672 30687 hybrid_clock.cc:648] HybridClock initialized: now 1781837400082631 us; error 62 us; skew 500 ppm
I20260619 02:50:00.083052 30687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:00.086462 30687 init.cc:377] Logged in from keytab as kudu/127.7.246.124@KRBTEST.COM (short username kudu)
I20260619 02:50:00.088012 30687 webserver.cc:460] Webserver started at http://127.7.246.124:43869/ using document root <none> and password file <none>
I20260619 02:50:00.088740 30687 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:00.088883 30687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:00.089244 30687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:00.092556 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data/instance:
uuid: "aef99197a4664cb5b70024da91b23e7e"
format_stamp: "Formatted at 2026-06-19 02:50:00 on dist-test-slave-9tkz"
server_key: "2b226f762081dd556517f86e70fcf255"
server_key_iv: "10c7b18d468e58ba799d8b307a4028e2"
server_key_version: "encryptionkey@0"
I20260619 02:50:00.092789 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 3
I20260619 02:50:00.093511 30687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal/instance:
uuid: "aef99197a4664cb5b70024da91b23e7e"
format_stamp: "Formatted at 2026-06-19 02:50:00 on dist-test-slave-9tkz"
server_key: "2b226f762081dd556517f86e70fcf255"
server_key_iv: "10c7b18d468e58ba799d8b307a4028e2"
server_key_version: "encryptionkey@0"
I20260619 02:50:00.093930 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.093403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35136)
0619 02:50:00.093651 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.093773 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:50:00.094903 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.094548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35152)
0619 02:50:00.094722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.094806 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 02:50:00.095106 30606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:33853: Network error: Client connection negotiation failed: client connection to 127.7.246.124:33853: connect: Connection refused (error 111)
I20260619 02:50:00.099572 30687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 02:50:00.103489 30702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:00.104928 30687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:50:00.105190 30687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "aef99197a4664cb5b70024da91b23e7e"
format_stamp: "Formatted at 2026-06-19 02:50:00 on dist-test-slave-9tkz"
server_key: "2b226f762081dd556517f86e70fcf255"
server_key_iv: "10c7b18d468e58ba799d8b307a4028e2"
server_key_version: "encryptionkey@0"
I20260619 02:50:00.105441 30687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:00.126238 30687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:00.130326 30687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:00.130646 30687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:00.151445 30753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.124:33853 every 8 connection(s)
I20260619 02:50:00.151469 30687 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.124:33853
I20260619 02:50:00.152861 30687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
I20260619 02:50:00.156561 30754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:00.157366 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30687
I20260619 02:50:00.157645 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837047186258-8153-0/minicluster-data/master-2/wal/instance
I20260619 02:50:00.158067 8153 external_mini_cluster.cc:1428] Setting key 0108455c0aabf77f4f3dd2445ad6d87f
I20260619 02:50:00.165921 30754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:50:00.182922 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.160001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.7.246.124:33853)
0619 02:50:00.160949 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.160968 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.161312 (+ 344us) server_negotiation.cc:407] Connection header received
0619 02:50:00.163339 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.163363 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.163779 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.164205 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.165631 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.167006 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.168267 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.169436 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.169647 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.174428 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.174439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.174454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.174505 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.178086 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.178848 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.178856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.178866 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.179051 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.179653 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.179658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.179662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.180463 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.180769 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.181171 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.181549 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":419,"threads_started":1}
I20260619 02:50:00.184741 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.168443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:43021 (local address 127.0.0.1:47096)
0619 02:50:00.169257 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.169305 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.169460 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.169918 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.169922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.170492 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.170730 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.170738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.172220 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.172224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.173294 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.173306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.174446 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.174452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.174707 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.175397 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.175420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.178207 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:00.182052 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.182063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.182071 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.182607 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.183051 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.183056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.183059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.183205 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.183741 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:00.183748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:00.183945 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 02:50:00.184162 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 02:50:00.184438 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":410,"threads_started":1}
I20260619 02:50:00.185128 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.168508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.7.246.126:43021)
0619 02:50:00.168748 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.168757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.169419 (+ 662us) server_negotiation.cc:407] Connection header received
0619 02:50:00.169637 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.169643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.169738 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.169848 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.170885 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.172046 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.173438 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.174211 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.174323 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.178400 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.178405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.178409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.178440 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.181923 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.182757 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.182761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.182763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.182906 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.183344 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.183349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.183351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.183597 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.183700 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.184225 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.184486 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:50:00.186417 30754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:50:00.201642 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.187013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:43175 (local address 127.0.0.1:38338)
0619 02:50:00.187176 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.187194 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.187286 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.187605 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.187612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.188055 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.188245 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.188256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.189661 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.189669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.191086 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.191101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.192214 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.192222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.192402 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.193109 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.193136 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.195654 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:00.198983 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.198990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.198994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.199510 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.199969 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.199975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.199979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.200146 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.200655 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:00.200660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:00.200837 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 02:50:00.201005 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:50:00.201246 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 02:50:00.201719 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.187118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.7.246.125:43175)
0619 02:50:00.187271 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.187275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.187290 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:50:00.187410 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.187414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.187468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.187541 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.188410 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.189461 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.191241 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.191996 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.192119 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.195854 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.195859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.195862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.195895 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.198854 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.199626 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.199630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.199632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.199847 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.200256 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.200259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.200261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.200515 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.200632 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.201129 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.201348 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:50:00.202734 30754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:00.203140 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.186284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.7.246.126:43021)
0619 02:50:00.186448 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.186454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.186572 (+ 118us) server_negotiation.cc:407] Connection header received
0619 02:50:00.186733 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.186738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.186811 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.186914 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.188411 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.189493 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.191354 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.191996 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.192119 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.196987 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.196992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.196995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.197025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.200224 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.200931 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.200935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.200937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.201118 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.201567 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.201572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.201576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.201983 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.202286 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.202580 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.202850 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837400, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:50:00.217845 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.203673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.7.246.124:33853)
0619 02:50:00.203841 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.203846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.203928 (+ 82us) server_negotiation.cc:407] Connection header received
0619 02:50:00.204105 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.204110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.204180 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.204288 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.205228 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.206384 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.207992 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.208635 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.208780 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.212241 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.212247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.212251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.212285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.215088 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.215947 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.215951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.215953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.216106 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.216557 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.216561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.216563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.216821 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.216928 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.217370 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.217535 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:50:00.218230 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.203705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35172)
0619 02:50:00.203854 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.203874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.203983 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.204337 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.204342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.204899 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.205072 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.205084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.206519 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.206523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.207835 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.207850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.208862 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.208874 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.209070 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.209703 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.209724 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.212052 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:00.215266 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.215273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.215279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.215815 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.216264 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.216270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.216273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.216439 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.216951 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:00.216956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:00.217111 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 02:50:00.217287 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:50:00.217805 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":155}
I20260619 02:50:00.220606 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.204963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.7.246.125:43175)
0619 02:50:00.205123 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.205130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.205150 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:50:00.205253 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.205264 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.205314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.205382 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.206817 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.207910 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.209507 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.210234 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.210363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.214574 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.214580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.214583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.214616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.217881 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.218741 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.218745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.218747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.218917 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.219295 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.219300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.219304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.219695 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.219939 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.220226 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.220395 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:50:00.224578 30754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: Bootstrap starting.
I20260619 02:50:00.225934 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:50:00.228642 30754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:00.229892 30754 log.cc:824] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:00.229360 30581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "886eedf066c2428d87708b4c90b93b70"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47104
I20260619 02:50:00.232817 30754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: No bootstrap required, opened a new log
I20260619 02:50:00.240000 30754 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.240360 30754 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef99197a4664cb5b70024da91b23e7e, State: Initialized, Role: FOLLOWER
I20260619 02:50:00.240835 30754 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.241789 30760 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.242183 30760 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.242830 30754 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:50:00.252332 30771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:00.252483 30771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:00.250665 30687 master_runner.cc:186] Error getting master registration for 127.7.246.126:43021: OK, instance_id { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" instance_seqno: 1781837399134416 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:50:00.261379 30687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:50:00.272328 30606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 4
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:50:00.286492 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.273190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35180)
0619 02:50:00.273391 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.273413 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.273538 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.273922 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.273926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.274358 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.274512 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.274521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.275591 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.275597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.276553 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.276564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.277355 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.277361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.277608 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.278260 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.278278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.280412 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:00.283800 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.283805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.283809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.284190 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.284583 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.284588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.284592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.284740 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.285414 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:00.285419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:00.285633 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:50:00.285793 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 02:50:00.286090 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:50:00.286490 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.273318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.7.246.124:33853)
0619 02:50:00.273505 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.273511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.273531 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:50:00.273700 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.273709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.273785 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.273887 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.274668 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.275433 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.276699 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.277188 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.277333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.280749 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.280758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.280763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.280805 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.283635 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.284324 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.284330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.284333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.284457 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.284861 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.284867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.284870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.285263 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.285438 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.285907 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.286114 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:50:00.292490 30606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: Bootstrap starting.
I20260619 02:50:00.295706 30606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:00.297014 30606 log.cc:824] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:00.300205 30606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: No bootstrap required, opened a new log
I20260619 02:50:00.305254 30606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.305639 30606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886eedf066c2428d87708b4c90b93b70, State: Initialized, Role: FOLLOWER
I20260619 02:50:00.306247 30606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.307253 30774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.307634 30774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.308568 30606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:00.308825 30581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:50:00.309090 30581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:00.312719 30581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.315554 30581 leader_election.cc:288] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [CANDIDATE]: Term 1 election: Requested vote from peers 444681890f254c7a87d5ae24b22d5490 (127.7.246.125:43175), aef99197a4664cb5b70024da91b23e7e (127.7.246.124:33853)
I20260619 02:50:00.315696 30729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "886eedf066c2428d87708b4c90b93b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aef99197a4664cb5b70024da91b23e7e"
I20260619 02:50:00.316229 30729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:00.320200 30729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886eedf066c2428d87708b4c90b93b70 in term 1.
I20260619 02:50:00.321113 30558 leader_election.cc:302] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 886eedf066c2428d87708b4c90b93b70, aef99197a4664cb5b70024da91b23e7e; no voters:
I20260619 02:50:00.321713 30774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:00.330034 30774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [term 1 LEADER]: Becoming Leader. State: Replica: 886eedf066c2428d87708b4c90b93b70, State: Running, Role: LEADER
I20260619 02:50:00.331028 30774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.342382 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886eedf066c2428d87708b4c90b93b70. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.342885 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:00.344023 30793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:00.356541 30680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } attempt: 5
I20260619 02:50:00.368479 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.340186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.7.246.124:33853)
0619 02:50:00.340413 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.340421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.343102 (+ 2681us) server_negotiation.cc:407] Connection header received
0619 02:50:00.343406 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.343411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.343489 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.343561 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.346372 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.347564 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.350797 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.351471 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.351569 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.355640 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.355651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.355654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.355688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.361122 (+ 5434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.365904 (+ 4782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.365912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.365918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.366105 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.366568 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.366575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.366579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.366980 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.367225 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.367775 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.368043 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 02:50:00.373095 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.340664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.7.246.126:43021)
0619 02:50:00.340894 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.340902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.343230 (+ 2328us) server_negotiation.cc:407] Connection header received
0619 02:50:00.343302 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.343310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.343394 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.343495 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.349709 (+ 6214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.350860 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.352688 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.356604 (+ 3916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.356741 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.364186 (+ 7445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.364197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.364205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.364265 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.368512 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.370426 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.370432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.370435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.370635 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.371091 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.371098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.371104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.371448 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.371691 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.372307 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.372631 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 02:50:00.373778 30560 tablet.cc:2369] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:00.351450 30539 master_runner.cc:186] Error getting master registration for 127.7.246.125:43175: OK, instance_id { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" instance_seqno: 1781837399629855 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:50:00.376148 30539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:50:00.378425 30793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:50:00 dist-test-slave-9tkz krb5kdc[30526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:50:00.381433 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.345164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.7.246.125:43175)
0619 02:50:00.345369 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.345376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.346588 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 02:50:00.347347 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.347353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.347443 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.347527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.355118 (+ 7591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.356233 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.357985 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.358686 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.358779 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.371858 (+ 13079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.371865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.371870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.371909 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.376487 (+ 4578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.377359 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.377364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.377367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.377565 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.377988 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.377993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.377997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.378355 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.378642 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.380815 (+ 2173us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.381073 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:50:00.382737 30708 tablet.cc:2369] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:00.390726 30683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.357097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:33853 (local address 127.0.0.1:35196)
0619 02:50:00.357474 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:00.357492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:00.357624 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:00.366490 (+ 8866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:00.366497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:00.367011 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:00.367259 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.367274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.373667 (+ 6393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.373674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.375269 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:00.375287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.376402 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.376412 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:00.376656 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.377313 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:00.377352 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:00.380245 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:00.386392 (+ 6147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.386397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.386402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.386956 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.388890 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:00.388895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:00.388899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:00.389049 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:00.389686 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:00.389691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:00.389859 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 02:50:00.390032 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 02:50:00.390334 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260619 02:50:00.390770 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.357833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.7.246.124:33853)
0619 02:50:00.366079 (+ 8246us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.366087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.366114 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:50:00.366185 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.366192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.366282 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.367425 (+ 1143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.367700 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.369038 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.375498 (+ 6460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.376172 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.376366 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.380536 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:00.380544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:00.380580 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:00.380743 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:00.386233 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.387102 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.387108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.387111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.388767 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:00.389170 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:00.389178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:00.389182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:00.389522 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:00.389700 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.390124 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.390334 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8081,"thread_start_us":7868,"threads_started":1}
I20260619 02:50:00.400045 30680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: Bootstrap starting.
I20260619 02:50:00.405539 30680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:00.406873 30680 log.cc:824] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:00.410322 30680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: No bootstrap required, opened a new log
I20260619 02:50:00.410321 30729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [term 1 FOLLOWER]: Refusing update from remote peer 886eedf066c2428d87708b4c90b93b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:00.411520 30774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:00.425163 30680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.425690 30680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444681890f254c7a87d5ae24b22d5490, State: Initialized, Role: FOLLOWER
I20260619 02:50:00.426298 30680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } }
I20260619 02:50:00.432811 30680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:00.434504 30654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:00.434476 30655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "886eedf066c2428d87708b4c90b93b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "444681890f254c7a87d5ae24b22d5490"
I20260619 02:50:00.435180 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.435797 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.438418 30654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [term 1 FOLLOWER]: Refusing update from remote peer 886eedf066c2428d87708b4c90b93b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:00.438969 30760 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886eedf066c2428d87708b4c90b93b70. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.439272 30760 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.441743 30655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 886eedf066c2428d87708b4c90b93b70 in term 1.
I20260619 02:50:00.443410 30798 mvcc.cc:205] Tried to move back new op lower bound from 7298405992070074368 to 7298405991763599360. Current Snapshot: MvccSnapshot[applied={T|T < 7298405992070074368}]
I20260619 02:50:00.444047 30774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:00.444624 30774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.444912 30774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:00.447333 30760 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.447594 30760 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.450452 30793 catalog_manager.cc:1347] Generated new cluster ID: e9130d86b88f4edd9cf9b1dde3299647
I20260619 02:50:00.450556 30793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:00.493979 30793 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:00.495885 30793 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:00.508603 30801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886eedf066c2428d87708b4c90b93b70. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.508930 30801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.509402 30816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.509452 30806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "886eedf066c2428d87708b4c90b93b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "886eedf066c2428d87708b4c90b93b70" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 43021 } } peers { permanent_uuid: "444681890f254c7a87d5ae24b22d5490" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 43175 } } peers { permanent_uuid: "aef99197a4664cb5b70024da91b23e7e" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 33853 } } }
I20260619 02:50:00.509851 30816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:00.509932 30806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:00.527175 30793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886eedf066c2428d87708b4c90b93b70: Generated new TSK 0
I20260619 02:50:00.528319 30793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:00.552872 30821 catalog_manager.cc:1259] Loaded cluster ID: e9130d86b88f4edd9cf9b1dde3299647
I20260619 02:50:00.552985 30821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: loading cluster ID for follower catalog manager: success
I20260619 02:50:00.556835 30821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: acquiring CA information for follower catalog manager: success
I20260619 02:50:00.559387 30821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 444681890f254c7a87d5ae24b22d5490: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:50:00.671614 30610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.652155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.7.246.126:43021)
0619 02:50:00.652361 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.652368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.665922 (+ 13554us) server_negotiation.cc:407] Connection header received
0619 02:50:00.669869 (+ 3947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.669878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.669999 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.670088 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.671355 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47124: BlockingRecv error: recv got EOF from 127.0.0.1:47124 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 02:50:00.672986 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.660413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.7.246.124:33853)
0619 02:50:00.660617 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.660627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.666404 (+ 5777us) server_negotiation.cc:407] Connection header received
0619 02:50:00.669127 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.669135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.669259 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.669346 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.672767 (+ 3421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35208: BlockingRecv error: recv got EOF from 127.0.0.1:35208 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 02:50:00.676311 30684 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.668055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.7.246.125:43175)
0619 02:50:00.668267 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.668273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.674222 (+ 5949us) server_negotiation.cc:407] Connection header received
0619 02:50:00.675053 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.675070 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.675174 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.675249 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.676089 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38372: BlockingRecv error: recv got EOF from 127.0.0.1:38372 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:50:00.740231 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.723786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.7.246.125:43175)
0619 02:50:00.723978 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.723984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.732124 (+ 8140us) server_negotiation.cc:407] Connection header received
0619 02:50:00.732405 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.732411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.732613 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.732782 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:00.733474 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.734703 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.736794 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.737836 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.737937 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.738728 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:00.739412 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:00.739539 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.739782 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.739879 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 02:50:00.783034 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.773529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.7.246.124:33853)
0619 02:50:00.773737 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.773744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.780686 (+ 6942us) server_negotiation.cc:407] Connection header received
0619 02:50:00.781240 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.781248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.781563 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.781655 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.782752 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35210: BlockingRecv error: recv got EOF from 127.0.0.1:35210 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:50:00.845119 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.827675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.7.246.125:43175)
0619 02:50:00.827888 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.827895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.838270 (+ 10375us) server_negotiation.cc:407] Connection header received
0619 02:50:00.838520 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.838526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.838642 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.838754 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:00.839447 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.840620 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.842682 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.843436 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.843544 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.844189 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:00.844369 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:00.844430 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.844664 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.844751 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 02:50:00.919972 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.910681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.7.246.124:33853)
0619 02:50:00.910957 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.910974 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.917972 (+ 6998us) server_negotiation.cc:407] Connection header received
0619 02:50:00.918511 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.918518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.918617 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.918695 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.919736 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35214: BlockingRecv error: recv got EOF from 127.0.0.1:35214 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260619 02:50:00.935046 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.927941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.7.246.125:43175)
0619 02:50:00.928110 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.928116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.928207 (+ 91us) server_negotiation.cc:407] Connection header received
0619 02:50:00.928517 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.928524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.928619 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.928720 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:00.929199 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.930569 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.932619 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.933280 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.933392 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.934087 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:00.934269 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:00.934341 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.934585 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.934688 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 02:50:00.950102 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.947587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.7.246.124:33853)
0619 02:50:00.947759 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.947765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.948315 (+ 550us) server_negotiation.cc:407] Connection header received
0619 02:50:00.948629 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.948636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.948734 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.948806 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.949851 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35226: BlockingRecv error: recv got EOF from 127.0.0.1:35226 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:50:00.965749 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.958514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38400 (local address 127.7.246.125:43175)
0619 02:50:00.958753 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.958760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.959318 (+ 558us) server_negotiation.cc:407] Connection header received
0619 02:50:00.959533 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.959539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.959646 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.959767 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:00.960277 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.961480 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.963507 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.964229 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.964376 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.964860 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:00.964998 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:00.965091 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.965279 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.965375 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260619 02:50:00.974917 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:00.972589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.7.246.124:33853)
0619 02:50:00.972798 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.972805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.973282 (+ 477us) server_negotiation.cc:407] Connection header received
0619 02:50:00.973568 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.973574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.973674 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.973743 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:00.974674 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35242: BlockingRecv error: recv got EOF from 127.0.0.1:35242 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 02:50:00.995069 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:00.987826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.7.246.125:43175)
0619 02:50:00.988035 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:00.988043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:00.988582 (+ 539us) server_negotiation.cc:407] Connection header received
0619 02:50:00.988947 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:00.988954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:00.989047 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:00.989152 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:00.989678 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.990843 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.992835 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:00.993608 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:00.993735 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:00.994233 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:00.994390 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:00.994490 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:00.994659 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:50:00.994742 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 02:50:01.006726 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.003521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.7.246.124:33853)
0619 02:50:01.003715 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.003721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.004960 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 02:50:01.005240 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.005245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.005335 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.005402 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.006486 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35256: BlockingRecv error: recv got EOF from 127.0.0.1:35256 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:50:01.033398 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.026296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.7.246.125:43175)
0619 02:50:01.026472 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.026479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.027021 (+ 542us) server_negotiation.cc:407] Connection header received
0619 02:50:01.027350 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.027357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.027449 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.027565 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.028047 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.029283 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.031250 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.031932 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.032069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.032580 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.032720 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.032812 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.032994 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.033084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 02:50:01.041730 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.039437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.7.246.124:33853)
0619 02:50:01.039634 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.039641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.040131 (+ 490us) server_negotiation.cc:407] Connection header received
0619 02:50:01.040312 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.040316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.040426 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.040517 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.041479 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35270: BlockingRecv error: recv got EOF from 127.0.0.1:35270 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:50:01.083928 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.077725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.7.246.125:43175)
0619 02:50:01.077989 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.077996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.078312 (+ 316us) server_negotiation.cc:407] Connection header received
0619 02:50:01.078576 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.078580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.078647 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.078746 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.079227 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.079995 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.082015 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.082530 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.082626 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.083115 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.083241 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.083309 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.083499 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.083561 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 02:50:01.093128 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.090910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.7.246.124:33853)
0619 02:50:01.091100 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.091106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.091668 (+ 562us) server_negotiation.cc:407] Connection header received
0619 02:50:01.091846 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.091851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.091943 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.092026 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.092935 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35276: BlockingRecv error: recv got EOF from 127.0.0.1:35276 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:50:01.165347 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.160190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.7.246.125:43175)
0619 02:50:01.160403 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.160410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.160774 (+ 364us) server_negotiation.cc:407] Connection header received
0619 02:50:01.161001 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.161007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.161082 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.161187 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.161624 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.162358 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.163643 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.164076 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.164222 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.164632 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.164742 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.164848 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.165007 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.165084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:50:01.173138 30797 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:01.171052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.7.246.124:33853)
0619 02:50:01.171294 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.171300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.171633 (+ 333us) server_negotiation.cc:407] Connection header received
0619 02:50:01.171891 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.171897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.171992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.172115 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.172928 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:50:01.254400 30771 catalog_manager.cc:1259] Loaded cluster ID: e9130d86b88f4edd9cf9b1dde3299647
I20260619 02:50:01.254500 30771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: loading cluster ID for follower catalog manager: success
I20260619 02:50:01.257023 30771 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: acquiring CA information for follower catalog manager: success
I20260619 02:50:01.258817 30771 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aef99197a4664cb5b70024da91b23e7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:50:01.311059 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.305190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.7.246.125:43175)
0619 02:50:01.305428 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.305434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.305735 (+ 301us) server_negotiation.cc:407] Connection header received
0619 02:50:01.305921 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.305925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.305993 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.306072 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.306469 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.307124 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.309250 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.309746 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.309891 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.310373 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.310484 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.310592 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.310708 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.310787 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 02:50:01.322762 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.316978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.7.246.124:33853)
0619 02:50:01.317264 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.317270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.317705 (+ 435us) server_negotiation.cc:407] Connection header received
0619 02:50:01.317863 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.317867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.318015 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.318111 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.318524 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.319227 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.320543 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.320949 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.321039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.321536 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.322097 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.322211 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.322359 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.322427 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 02:50:01.337311 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.329339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.7.246.124:33853)
0619 02:50:01.329563 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.329571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.330956 (+ 1385us) server_negotiation.cc:407] Connection header received
0619 02:50:01.331174 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.331182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.331316 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.331430 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.332135 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.332968 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.335131 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.335782 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.335866 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.336489 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.336624 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.336684 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.336916 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.336996 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:50:01.337703 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.330241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.7.246.125:43175)
0619 02:50:01.330451 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.330457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.331635 (+ 1178us) server_negotiation.cc:407] Connection header received
0619 02:50:01.331849 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.331857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.331958 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.332047 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.332816 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.333820 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.335785 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.336291 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.336440 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.336888 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.337021 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.337106 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.337305 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.337391 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:50:01.338662 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.329197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.7.246.126:43021)
0619 02:50:01.332155 (+ 2958us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.332163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.332182 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:50:01.332296 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.332304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.332529 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.332800 (+ 271us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:01.333410 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.334429 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.335979 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.336583 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.336715 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.337290 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:01.338001 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:01.338132 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.338256 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.338336 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2839,"thread_start_us":2732,"threads_started":1}
I20260619 02:50:01.343410 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30539
I20260619 02:50:01.344784 30601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:01.455265 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30539
W20260619 02:50:01.475665 30704 connection.cc:535] server connection from 127.0.0.1:35180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:50:01.476244 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30613
I20260619 02:50:01.477289 30675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:01.586452 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30613
I20260619 02:50:01.606945 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30687
I20260619 02:50:01.608124 30749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:01.715157 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30687
2026-06-19T02:50:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4369 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:01 dist-test-slave-9tkz krb5kdc[30935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:01 dist-test-slave-9tkz krb5kdc[30935](info): set up 2 sockets
Jun 19 02:50:01 dist-test-slave-9tkz krb5kdc[30935](info): commencing operation
krb5kdc: starting...
W20260619 02:50:02.822428 8153 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/test-user.keytab.
Jun 19 02:50:02 dist-test-slave-9tkz krb5kdc[30935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.7.246.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
Entry for principal kudu/127.7.246.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab.
I20260619 02:50:03.084177 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:39341
--webserver_interface=127.7.246.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:03.413282 30948 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:50:03.413666 30948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:03.413753 30948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:03.413834 30948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:03.421003 30948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:03.421085 30948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:03.421140 30948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:03.421195 30948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:03.421249 30948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:03.430783 30948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.126.keytab
--principal=kudu/127.7.246.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.126:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.246.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:50:03.431980 30948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:03.433225 30948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:03.445369 30956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:03.445407 30953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:03.445418 30954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:03.445835 30948 server_base.cc:1034] running on GCE node
I20260619 02:50:03.446910 30948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:03.448340 30948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:03.449626 30948 hybrid_clock.cc:648] HybridClock initialized: now 1781837403449513 us; error 53 us; skew 500 ppm
I20260619 02:50:03.449952 30948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:50:03 dist-test-slave-9tkz krb5kdc[30935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:03.459995 30948 init.cc:377] Logged in from keytab as kudu/127.7.246.126@KRBTEST.COM (short username kudu)
I20260619 02:50:03.461520 30948 webserver.cc:460] Webserver started at http://127.7.246.126:45393/ using document root <none> and password file <none>
I20260619 02:50:03.462301 30948 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:03.462436 30948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:03.462761 30948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:03.465766 30948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data/instance:
uuid: "3099a10eff3547248c982c7c48f0b5a3"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "7944ec1301117949e2683e209c4d1d32"
server_key_iv: "2b2078d5193756a95120d9fa3b761a9b"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.466580 30948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance:
uuid: "3099a10eff3547248c982c7c48f0b5a3"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "7944ec1301117949e2683e209c4d1d32"
server_key_iv: "2b2078d5193756a95120d9fa3b761a9b"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.471784 30948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:50:03.475243 30963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:03.476466 30948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:50:03.476656 30948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal
uuid: "3099a10eff3547248c982c7c48f0b5a3"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "7944ec1301117949e2683e209c4d1d32"
server_key_iv: "2b2078d5193756a95120d9fa3b761a9b"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.476879 30948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:03.519318 30948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:03.523450 30948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:03.523738 30948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:03.542402 30948 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.126:39341
I20260619 02:50:03.542410 31014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.126:39341 every 8 connection(s)
I20260619 02:50:03.543812 30948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:03.547610 31015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:03.549401 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 30948
I20260619 02:50:03.549635 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-0/wal/instance
I20260619 02:50:03.550060 8153 external_mini_cluster.cc:1428] Setting key 536ec6392b3b5363c842140ab6673718
I20260619 02:50:03.554167 31015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:03 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.7.246.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.125@KRBTEST.COM" created.
I20260619 02:50:03.578503 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.558909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.7.246.126:39341)
0619 02:50:03.559600 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:03.559611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:03.559659 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:50:03.560906 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:03.560916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:03.561224 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:03.561437 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:03.562610 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.564498 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.565978 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.566950 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.567144 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.570989 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:03.570997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:03.571008 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:03.571058 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:03.573954 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:03.574957 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:03.574964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:03.574966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:03.575152 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:03.575548 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:03.575552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:03.575556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:03.576105 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:03.576301 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:03.576881 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 02:50:03.577279 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":467,"thread_start_us":303,"threads_started":1}
I20260619 02:50:03.578542 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.557546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39341 (local address 127.0.0.1:43516)
0619 02:50:03.558457 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.558513 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:03.560498 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:03.561452 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:03.561457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:03.562061 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:03.562397 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.562413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.564668 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.564671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.565838 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:03.565849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:03.567164 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:03.567169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:03.567306 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:03.567978 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:03.568001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:03.570682 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:03.574104 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:03.574121 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:03.574157 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:03.574806 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:03.575265 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:03.575272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:03.575276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:03.575434 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:03.576259 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:03.576265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:03.576486 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 02:50:03.576764 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 02:50:03.577145 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":364,"threads_started":1}
I20260619 02:50:03.581893 31015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:03.583112 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.582602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39322)
0619 02:50:03.582817 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.582952 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 02:50:03.583425 30965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
I20260619 02:50:03.585155 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.584649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39336)
0619 02:50:03.584856 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.584962 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:50:03.585353 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:33259: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
Entry for principal kudu/127.7.246.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab.
I20260619 02:50:03.609440 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:33259
--webserver_interface=127.7.246.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:50:03.634735 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } attempt: 1
I20260619 02:50:03.635978 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.635477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39346)
0619 02:50:03.635674 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.635816 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:50:03.637063 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.636709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39352)
0619 02:50:03.636854 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.636926 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:50:03.637259 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:33259: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
I20260619 02:50:03.705518 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } attempt: 2
I20260619 02:50:03.706964 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.706483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39354)
0619 02:50:03.706733 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.706827 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 02:50:03.708014 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.707634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39364)
0619 02:50:03.707808 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.707883 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 02:50:03.708195 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:33259: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
I20260619 02:50:03.799440 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } attempt: 3
I20260619 02:50:03.800534 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.800110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39370)
0619 02:50:03.800311 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.800405 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 02:50:03.801534 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.801201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39378)
0619 02:50:03.801344 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.801419 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:50:03.801762 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:33259: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
W20260619 02:50:03.933413 31022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:50:03.933787 31022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:03.933907 31022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:03.934063 31022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:03.941154 31022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:03.941298 31022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:03.941401 31022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:03.941504 31022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:03.941609 31022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:03.945008 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } attempt: 4
I20260619 02:50:03.946437 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.945719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39394)
0619 02:50:03.946116 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.946242 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 02:50:03.947679 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.947302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39406)
0619 02:50:03.947464 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.947558 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 02:50:03.947906 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.125:33259: Network error: Client connection negotiation failed: client connection to 127.7.246.125:33259: connect: Connection refused (error 111)
I20260619 02:50:03.953212 31022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.125.keytab
--principal=kudu/127.7.246.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.125:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.7.246.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:50:03.954545 31022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:03.955847 31022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:03.968086 31027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:03.968283 31030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:03.969249 31028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:03.969636 31022 server_base.cc:1034] running on GCE node
I20260619 02:50:03.970430 31022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:03.971832 31022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:03.973080 31022 hybrid_clock.cc:648] HybridClock initialized: now 1781837403973020 us; error 38 us; skew 500 ppm
I20260619 02:50:03.973434 31022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:50:03 dist-test-slave-9tkz krb5kdc[30935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:03.976701 31022 init.cc:377] Logged in from keytab as kudu/127.7.246.125@KRBTEST.COM (short username kudu)
I20260619 02:50:03.978144 31022 webserver.cc:460] Webserver started at http://127.7.246.125:39707/ using document root <none> and password file <none>
I20260619 02:50:03.978828 31022 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:03.978914 31022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:03.979218 31022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:03.982311 31022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data/instance:
uuid: "1c67525e8968444da01eeb12c78809f1"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "cdb89f041a231771677b540a533b7b76"
server_key_iv: "08f433fcd6635d360946499e76dae401"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.983052 31022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal/instance:
uuid: "1c67525e8968444da01eeb12c78809f1"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "cdb89f041a231771677b540a533b7b76"
server_key_iv: "08f433fcd6635d360946499e76dae401"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.988125 31022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:50:03.991663 31037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:03.992983 31022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:50:03.993124 31022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal
uuid: "1c67525e8968444da01eeb12c78809f1"
format_stamp: "Formatted at 2026-06-19 02:50:03 on dist-test-slave-9tkz"
server_key: "cdb89f041a231771677b540a533b7b76"
server_key_iv: "08f433fcd6635d360946499e76dae401"
server_key_version: "encryptionkey@0"
I20260619 02:50:03.993304 31022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:04.011991 31022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:04.015934 31022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:04.016156 31022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:04.034438 31022 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.125:33259
I20260619 02:50:04.034456 31088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.125:33259 every 8 connection(s)
I20260619 02:50:04.035781 31022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/data/info.pb
I20260619 02:50:04.039419 31089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:04.043730 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 31022
I20260619 02:50:04.043920 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-1/wal/instance
I20260619 02:50:04.044472 8153 external_mini_cluster.cc:1428] Setting key e792b52e30093d5b4d517e207911515c
I20260619 02:50:04.047024 31089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:50:04.070485 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.050319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.7.246.126:39341)
0619 02:50:04.050548 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.050554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.051704 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 02:50:04.054539 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.054544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.054610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.054698 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.056555 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.057781 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.059250 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.059971 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.060111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.063669 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.063678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.063686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.063731 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.067440 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.068162 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.068169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.068173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.068302 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.068732 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.068738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.068743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.069024 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.069191 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.069921 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.070145 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 02:50:04.071429 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.050724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39341 (local address 127.0.0.1:43526)
0619 02:50:04.051527 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.051575 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.054243 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.054991 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.055005 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.055789 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.056390 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.056406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.057934 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.057937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.059088 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.059099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.060170 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.060175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.060362 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.061051 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.061073 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.063440 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.067582 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.067589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.067602 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.068032 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.068441 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.068445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.068447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.068620 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.069158 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.069168 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.069512 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.069851 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.070342 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for kudu/127.7.246.124@KRBTEST.COM; defaulting to no policy
I20260619 02:50:04.073175 31089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.7.246.124@KRBTEST.COM" created.
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:50:04.089071 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.074171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.7.246.125:33259)
0619 02:50:04.074859 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.074869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.074903 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:50:04.074958 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.074968 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.075205 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.075366 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.076018 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.077717 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.078846 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.079792 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.079940 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.083285 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.083294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.083307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.083333 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.086164 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.086996 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.087001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.087003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.087110 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.087524 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.087527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.087529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.087841 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.087990 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.088413 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.088745 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":497,"thread_start_us":347,"threads_started":1}
I20260619 02:50:04.089299 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.073972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39410)
0619 02:50:04.074154 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.074176 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.074270 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.075365 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.075368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.075711 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.075851 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.075859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.077850 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.077854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.078690 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.078700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.079971 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.079975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.080158 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.080816 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.080833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.083079 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.086308 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.086316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.086323 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.086855 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.087229 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.087237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.087240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.087423 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.087970 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.087978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.088139 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.088295 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.088632 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:50:04.091917 31089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:04.093168 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.092734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48316)
0619 02:50:04.092916 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.093013 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 02:50:04.093580 31041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.095139 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.094779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48330)
0619 02:50:04.094916 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.094986 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:50:04.095395 31089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
Entry for principal kudu/127.7.246.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab.
I20260619 02:50:04.108736 8153 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
/tmp/dist-test-taskfqHoa6/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:46559
--webserver_interface=127.7.246.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.7.246.84:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:50:04.144827 31089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 1
I20260619 02:50:04.145927 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.145468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48338)
0619 02:50:04.145674 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.145778 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 02:50:04.147028 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.146569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48348)
0619 02:50:04.146796 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.146893 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 02:50:04.147274 31089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.215515 31089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 2
I20260619 02:50:04.216796 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.216160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48352)
0619 02:50:04.216464 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.216596 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 02:50:04.217979 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.217526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48364)
0619 02:50:04.217738 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.217833 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 02:50:04.218226 31089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.247124 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } attempt: 5
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:50:04.260310 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.247974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.7.246.125:33259)
0619 02:50:04.248157 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.248163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.248184 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:50:04.248381 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.248387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.248470 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.248575 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.249333 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.250088 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.251240 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.251690 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.251819 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.255093 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.255098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.255101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.255131 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.257789 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.258432 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.258439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.258443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.258566 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.258925 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.258931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.258935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.259208 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.259364 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.259767 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.259992 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:50:04.260310 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.247862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39426)
0619 02:50:04.248074 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.248094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.248227 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.248604 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.248608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.249007 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.249173 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.249181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.250218 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.250225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.251092 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.251102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.251842 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.251848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.252061 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.252681 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.252698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.254898 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.257915 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.257921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.257925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.258272 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.258678 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.258683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.258687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.258817 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.259329 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.259334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.259500 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.259666 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.259958 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 02:50:04.261675 31015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:04.263005 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.262456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48370)
0619 02:50:04.262658 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.262774 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 02:50:04.264163 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.263646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48380)
0619 02:50:04.263863 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.263989 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 02:50:04.264430 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.309460 31089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 3
I20260619 02:50:04.310716 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.310135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48386)
0619 02:50:04.310393 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.310518 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 02:50:04.311873 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.311387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48390)
0619 02:50:04.311636 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.311730 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 02:50:04.312120 31089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.315677 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 1
I20260619 02:50:04.316807 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.316245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48404)
0619 02:50:04.316472 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.316591 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 02:50:04.317927 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.317448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48418)
0619 02:50:04.317665 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.317765 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 02:50:04.318213 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.386487 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 2
I20260619 02:50:04.387749 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.387164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48434)
0619 02:50:04.387429 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.387544 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 02:50:04.388780 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.388381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48450)
0619 02:50:04.388567 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.388654 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 02:50:04.389019 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
W20260619 02:50:04.439023 31096 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:50:04.439384 31096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:04.439507 31096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:04.439653 31096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:04.447273 31096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:04.447362 31096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:04.447419 31096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:04.447499 31096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:04.447618 31096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:04.455344 31089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 4
I20260619 02:50:04.456502 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.455969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48454)
0619 02:50:04.456212 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.456317 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 02:50:04.457479 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.457129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48458)
0619 02:50:04.457277 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.457356 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:50:04.457803 31089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.458544 31096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.246.84:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.7.246.126:39341,127.7.246.125:33259,127.7.246.124:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/krb5kdc/kudu_127.7.246.124.keytab
--principal=kudu/127.7.246.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.246.124:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.7.246.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:50:04.459825 31096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:04.461087 31096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:04.472817 31101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:04.472932 31102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:04.473495 31096 server_base.cc:1034] running on GCE node
W20260619 02:50:04.473580 31104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:04.474536 31096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:04.475849 31096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:04.477056 31096 hybrid_clock.cc:648] HybridClock initialized: now 1781837404477001 us; error 32 us; skew 500 ppm
I20260619 02:50:04.477353 31096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:04.480319 31096 init.cc:377] Logged in from keytab as kudu/127.7.246.124@KRBTEST.COM (short username kudu)
I20260619 02:50:04.481886 31096 webserver.cc:460] Webserver started at http://127.7.246.124:32777/ using document root <none> and password file <none>
I20260619 02:50:04.482497 31096 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:04.482580 31096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:04.482820 31096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:04.485608 31096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data/instance:
uuid: "6c669725a9474d208f4f5b195a9b9d6d"
format_stamp: "Formatted at 2026-06-19 02:50:04 on dist-test-slave-9tkz"
server_key: "11a17e43779e7549729346e9db3741f5"
server_key_iv: "78a24686e34e499e3b652986f105dea5"
server_key_version: "encryptionkey@0"
I20260619 02:50:04.486377 31096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal/instance:
uuid: "6c669725a9474d208f4f5b195a9b9d6d"
format_stamp: "Formatted at 2026-06-19 02:50:04 on dist-test-slave-9tkz"
server_key: "11a17e43779e7549729346e9db3741f5"
server_key_iv: "78a24686e34e499e3b652986f105dea5"
server_key_version: "encryptionkey@0"
I20260619 02:50:04.491289 31096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:50:04.494777 31111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:04.495234 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 3
I20260619 02:50:04.496497 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.495966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48464)
0619 02:50:04.496177 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.496279 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:50:04.496498 31096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:50:04.496681 31096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data,/tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal
uuid: "6c669725a9474d208f4f5b195a9b9d6d"
format_stamp: "Formatted at 2026-06-19 02:50:04 on dist-test-slave-9tkz"
server_key: "11a17e43779e7549729346e9db3741f5"
server_key_iv: "78a24686e34e499e3b652986f105dea5"
server_key_version: "encryptionkey@0"
I20260619 02:50:04.497025 31096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:04.497742 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.497212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48474)
0619 02:50:04.497484 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.497602 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260619 02:50:04.497988 31015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.246.124:46559: Network error: Client connection negotiation failed: client connection to 127.7.246.124:46559: connect: Connection refused (error 111)
I20260619 02:50:04.530064 31096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:04.534418 31096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:04.534708 31096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:04.557827 31096 rpc_server.cc:307] RPC server started. Bound to: 127.7.246.124:46559
I20260619 02:50:04.557863 31162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.246.124:46559 every 8 connection(s)
I20260619 02:50:04.559240 31096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/data/info.pb
I20260619 02:50:04.563066 8153 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu as pid 31096
I20260619 02:50:04.563223 31163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:04.563259 8153 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfqHoa6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837047186258-8153-0/minicluster-data/master-2/wal/instance
I20260619 02:50:04.563797 8153 external_mini_cluster.cc:1428] Setting key 3b8b54695db45f6358b96cc3f11d6bdf
I20260619 02:50:04.573036 31163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:50:04.588478 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.565632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.7.246.124:46559)
0619 02:50:04.566449 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.566461 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.566501 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:50:04.568041 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.568064 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.568447 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.568785 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.570590 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.572032 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.573808 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.574525 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.574651 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.579214 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.579221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.579231 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.579263 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.583316 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.584278 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.584289 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.584301 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.584535 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.584981 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.584990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.584995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.585786 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.586069 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.586745 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.587180 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":374,"threads_started":1}
I20260619 02:50:04.602260 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.576023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.126:39341 (local address 127.0.0.1:43534)
0619 02:50:04.577207 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.577251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.577435 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.577857 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.577865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.578521 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.578758 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.578766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.579922 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.579929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.581197 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.581237 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.582511 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.582520 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.582986 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.583726 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.583749 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.595557 (+ 11808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.599590 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.599599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.599605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.600009 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.600483 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.600488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.600492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.600635 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.601266 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.601272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.601430 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.601678 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.601978 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":752,"threads_started":1}
I20260619 02:50:04.602841 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.576016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.7.246.126:39341)
0619 02:50:04.576582 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.576589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.577368 (+ 779us) server_negotiation.cc:407] Connection header received
0619 02:50:04.577605 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.577609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.577673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.577790 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.578946 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.579771 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.581436 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.582255 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.582397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.595815 (+ 13418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.595825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.595830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.595877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.599457 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.600130 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.600135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.600138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.600377 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.600761 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.600769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.600773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.601154 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.601297 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.601808 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.602048 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260619 02:50:04.604296 31163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.126@KRBTEST.COM
I20260619 02:50:04.611380 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.594389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.7.246.126:39341)
0619 02:50:04.595434 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.595439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.595466 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:50:04.595528 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.595534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.595609 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.595709 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.597529 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.598812 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.600312 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.600891 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.600984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.606028 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.606038 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.606041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.606068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.608793 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.609432 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.609436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.609441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.609625 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.610055 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.610062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.610066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.610355 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.610546 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.610882 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.611031 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":894,"thread_start_us":708,"threads_started":1}
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
I20260619 02:50:04.618530 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.605017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.125:33259 (local address 127.0.0.1:39438)
0619 02:50:04.605423 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.605441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.605572 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.605920 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.605925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.606256 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.606390 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.606399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.607774 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.607778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.608620 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.608630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.609411 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.609420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.609635 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.610205 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.610221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.612497 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.616199 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.616205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.616209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.616653 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.617057 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.617060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.617062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.617187 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.617827 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.617830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.617955 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.618065 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.618243 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
I20260619 02:50:04.618868 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.605085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.7.246.125:33259)
0619 02:50:04.605280 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.605287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.605521 (+ 234us) server_negotiation.cc:407] Connection header received
0619 02:50:04.605685 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.605688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.605756 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.605851 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.606541 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.607631 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.608765 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.609257 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.609346 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.612690 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.612697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.612701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.612750 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.616065 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.616768 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.616772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.616774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.616946 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.617299 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.617304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.617307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.617686 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.617831 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.618189 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.618447 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:50:04.619776 31163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.246.125@KRBTEST.COM
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837404, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.124@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:50:04.629853 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.613227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.7.246.125:33259)
0619 02:50:04.614030 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.614034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.614054 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:50:04.614093 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.614096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.614146 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.614225 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.615736 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.616810 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.618460 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.619102 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.619208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.624330 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.624335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.624339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.624367 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.627125 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.627857 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.627862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.627868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.628010 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.628372 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.628378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.628384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.628658 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.628853 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.629244 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.629431 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":509,"threads_started":1}
I20260619 02:50:04.634321 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.620655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.7.246.124:46559)
0619 02:50:04.620815 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.620819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.621019 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:50:04.621164 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.621167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.621218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.621296 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.622192 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.622828 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.624492 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.625163 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.625296 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.628431 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.628456 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.628459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.628491 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.631524 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.632328 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.632334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.632338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.632455 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.632832 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.632837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.632840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.633130 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.633293 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.633729 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.633996 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:50:04.634658 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.620672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48498)
0619 02:50:04.620931 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.620951 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.621066 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.621355 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.621364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.621856 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.622059 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.622071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.622966 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.622973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.624364 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.624375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.625330 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.625334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.625494 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.626071 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.626095 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.628239 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.631675 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.631683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.631687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.632204 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.632574 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.632578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.632582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.632720 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.633260 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.633264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.633413 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.633766 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.634363 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":273}
I20260619 02:50:04.635217 8153 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:50:04.639394 31163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: Bootstrap starting.
I20260619 02:50:04.637658 30990 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3099a10eff3547248c982c7c48f0b5a3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43546
I20260619 02:50:04.642458 31163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:04.643437 31163 log.cc:824] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:04.646083 31163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: No bootstrap required, opened a new log
I20260619 02:50:04.652774 31163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.653080 31163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c669725a9474d208f4f5b195a9b9d6d, State: Initialized, Role: FOLLOWER
I20260619 02:50:04.653505 31163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.654508 31172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.654896 31172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.655372 31163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:50:04.664371 31183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:04.664487 31183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:04.662132 31096 master_runner.cc:186] Error getting master registration for 127.7.246.126:39341: OK, instance_id { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" instance_seqno: 1781837403536971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:50:04.673161 31096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:50:04.675218 31015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 4
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.126@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
I20260619 02:50:04.690801 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.675910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.7.246.124:46559)
0619 02:50:04.676174 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.676179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.676200 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:50:04.676310 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.676314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.676387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.676494 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.677263 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.678129 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.679424 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.679933 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.680071 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.683993 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.684000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.684005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.684053 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.687739 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.688544 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.688549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.688552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.688688 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.689116 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.689121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.689124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.689611 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.689768 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.690231 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.690445 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 02:50:04.690797 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.675864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48500)
0619 02:50:04.676047 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.676068 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.676200 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.676529 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.676534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.676943 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.677106 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.677115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.678304 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.678310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.679266 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.679278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.680095 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.680102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.680314 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.680951 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.680996 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.683767 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.687904 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.687909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.687913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.688423 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.688815 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.688820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.688823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.689005 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.689743 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.689748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.689932 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.690100 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.690384 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:50:04.696636 31015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: Bootstrap starting.
I20260619 02:50:04.700654 31015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:04.701931 31015 log.cc:824] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:04.704670 31015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: No bootstrap required, opened a new log
I20260619 02:50:04.709964 31015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.710301 31015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3099a10eff3547248c982c7c48f0b5a3, State: Initialized, Role: FOLLOWER
I20260619 02:50:04.710732 31015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.711681 31186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.712100 31186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.712785 31015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:04.712920 30990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:50:04.713125 30990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:04.716501 30990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.719175 30990 leader_election.cc:288] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [CANDIDATE]: Term 1 election: Requested vote from peers 1c67525e8968444da01eeb12c78809f1 (127.7.246.125:33259), 6c669725a9474d208f4f5b195a9b9d6d (127.7.246.124:46559)
I20260619 02:50:04.719316 31138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3099a10eff3547248c982c7c48f0b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c669725a9474d208f4f5b195a9b9d6d"
I20260619 02:50:04.719822 31138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:04.723222 31138 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3099a10eff3547248c982c7c48f0b5a3 in term 1.
I20260619 02:50:04.723855 30967 leader_election.cc:302] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3099a10eff3547248c982c7c48f0b5a3, 6c669725a9474d208f4f5b195a9b9d6d; no voters:
I20260619 02:50:04.724318 31186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:04.735265 31186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3099a10eff3547248c982c7c48f0b5a3, State: Running, Role: LEADER
I20260619 02:50:04.735847 31186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.743031 31195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3099a10eff3547248c982c7c48f0b5a3. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.743465 31195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:04.747499 31198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:04.757031 31089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } attempt: 5
I20260619 02:50:04.761432 31198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:50:04 dist-test-slave-9tkz krb5kdc[30935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837403, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.246.125@KRBTEST.COM for kudu/127.7.246.124@KRBTEST.COM
W20260619 02:50:04.776510 30969 tablet.cc:2369] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:04.778723 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.758082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.246.124:46559 (local address 127.0.0.1:48510)
0619 02:50:04.758300 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.758323 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.758511 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.759242 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.759250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.759905 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:04.760180 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.760194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.762030 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.762039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.763887 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.763907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.765432 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.765441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.765772 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.766495 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:04.766525 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:04.770023 (+ 3498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:04.775172 (+ 5149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.775177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.775184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.775857 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.776370 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:04.776377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:04.776384 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:04.776575 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:04.777378 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:04.777386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:04.777602 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.777773 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.778152 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 02:50:04.779927 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.757768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.7.246.124:46559)
0619 02:50:04.757955 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.757960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.758685 (+ 725us) server_negotiation.cc:407] Connection header received
0619 02:50:04.758761 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.758770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.758884 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.758993 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.760417 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.761819 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.764099 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.765033 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.765176 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.770275 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.770285 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.770292 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.770342 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.774986 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.776019 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.776027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.776033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.776222 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.776702 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.776709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.776714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.777138 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.777309 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.779142 (+ 1833us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.779421 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 02:50:04.786821 31117 tablet.cc:2369] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:04.787003 31089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: Bootstrap starting.
I20260619 02:50:04.791450 31089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:04.794224 31138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [term 1 FOLLOWER]: Refusing update from remote peer 3099a10eff3547248c982c7c48f0b5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:04.795368 31195 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:04.795557 30948 master_runner.cc:186] Error getting master registration for 127.7.246.125:33259: OK, instance_id { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" instance_seqno: 1781837404029081 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:50:04.808368 30948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:50:04.809693 31089 log.cc:824] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:04.816354 31089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: No bootstrap required, opened a new log
I20260619 02:50:04.823184 31172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3099a10eff3547248c982c7c48f0b5a3. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.823510 31172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.828153 31089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.828536 31089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c67525e8968444da01eeb12c78809f1, State: Initialized, Role: FOLLOWER
I20260619 02:50:04.829172 31204 mvcc.cc:205] Tried to move back new op lower bound from 7298406009981173760 to 7298406009803870208. Current Snapshot: MvccSnapshot[applied={T|T < 7298406009981173760}]
I20260619 02:50:04.829025 31089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } }
I20260619 02:50:04.834077 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.783453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.7.246.125:33259)
0619 02:50:04.783622 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.783628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.797241 (+ 13613us) server_negotiation.cc:407] Connection header received
0619 02:50:04.797479 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.797486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.797598 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.797664 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.799871 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.800871 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.802961 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.803721 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.803836 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.820132 (+ 16296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.820139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.820146 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.820184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.824201 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.829122 (+ 4921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.829128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.829132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.829285 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.829898 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.829906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.829912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.830381 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.830615 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.833514 (+ 2899us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.833752 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:50:04.840095 31214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.842494 31214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.842355 31089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:04.842391 31064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3099a10eff3547248c982c7c48f0b5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1c67525e8968444da01eeb12c78809f1"
I20260619 02:50:04.842730 31063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:04.846658 31063 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [term 1 FOLLOWER]: Refusing update from remote peer 3099a10eff3547248c982c7c48f0b5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:04.850272 31064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3099a10eff3547248c982c7c48f0b5a3 in term 1.
I20260619 02:50:04.852398 31186 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:04.856120 31186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.856431 31186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:04.860949 31198 catalog_manager.cc:1347] Generated new cluster ID: 133417175d274a6ab2ea313369505147
I20260619 02:50:04.861043 31198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:04.862807 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.783616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.7.246.126:39341)
0619 02:50:04.783823 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.783831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.791007 (+ 7176us) server_negotiation.cc:407] Connection header received
0619 02:50:04.800535 (+ 9528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.800541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.800618 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.800732 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.825189 (+ 24457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.826306 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.836489 (+ 10183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.837153 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.837258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.843224 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.843231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.843235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.843278 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.846794 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.858221 (+ 11427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.858227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.858232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.858437 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.859799 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.859807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.859810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.860205 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.860481 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.862159 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.862433 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:50:04.864231 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.785708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.7.246.124:46559)
0619 02:50:04.785980 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.785987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.807170 (+ 21183us) server_negotiation.cc:407] Connection header received
0619 02:50:04.807639 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.807645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.807720 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.807785 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:04.817956 (+ 10171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.819104 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.831162 (+ 12058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.832125 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.832246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.847760 (+ 15514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:04.847768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:04.847774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:04.847817 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:04.853476 (+ 5659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.859142 (+ 5666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.859148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.859152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.859326 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:04.860955 (+ 1629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:04.860961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:04.860965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:04.861360 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:04.861617 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.863576 (+ 1959us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.863831 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:50:04.881305 31172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.881609 31172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.910744 31221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.911201 31221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.911540 31214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3099a10eff3547248c982c7c48f0b5a3. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.911790 31214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:04.913491 31195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3099a10eff3547248c982c7c48f0b5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3099a10eff3547248c982c7c48f0b5a3" member_type: VOTER last_known_addr { host: "127.7.246.126" port: 39341 } } peers { permanent_uuid: "1c67525e8968444da01eeb12c78809f1" member_type: VOTER last_known_addr { host: "127.7.246.125" port: 33259 } } peers { permanent_uuid: "6c669725a9474d208f4f5b195a9b9d6d" member_type: VOTER last_known_addr { host: "127.7.246.124" port: 46559 } } }
I20260619 02:50:04.916965 31195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:04.926095 31198 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:04.928428 31198 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:04.952364 31198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3099a10eff3547248c982c7c48f0b5a3: Generated new TSK 0
I20260619 02:50:04.953796 31198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:04.983331 31230 catalog_manager.cc:1259] Loaded cluster ID: 133417175d274a6ab2ea313369505147
I20260619 02:50:04.983487 31230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: loading cluster ID for follower catalog manager: success
I20260619 02:50:04.987358 31230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: acquiring CA information for follower catalog manager: success
I20260619 02:50:04.989666 31230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c67525e8968444da01eeb12c78809f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:50:05.108749 31169 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.094060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.7.246.126:39341)
0619 02:50:05.094264 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.094271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.105872 (+ 11601us) server_negotiation.cc:407] Connection header received
0619 02:50:05.106880 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.106886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.106989 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.107075 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.108392 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43572: BlockingRecv error: recv got EOF from 127.0.0.1:43572 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 02:50:05.111660 31170 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.092962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.7.246.125:33259)
0619 02:50:05.093171 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.093180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.103657 (+ 10477us) server_negotiation.cc:407] Connection header received
0619 02:50:05.109730 (+ 6073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.109741 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.109893 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.110001 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.111268 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39468: BlockingRecv error: recv got EOF from 127.0.0.1:39468 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 02:50:05.113157 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.100834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.7.246.124:46559)
0619 02:50:05.101048 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.101057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.110443 (+ 9386us) server_negotiation.cc:407] Connection header received
0619 02:50:05.111388 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.111407 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.111494 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.111573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.112845 (+ 1272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48528: BlockingRecv error: recv got EOF from 127.0.0.1:48528 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:50:05.159812 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.141702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.7.246.125:33259)
0619 02:50:05.141919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.141926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.150903 (+ 8977us) server_negotiation.cc:407] Connection header received
0619 02:50:05.151526 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.151532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.151839 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.152001 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.152821 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.153993 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.156293 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.157040 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.157143 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.158063 (+ 920us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.158884 (+ 821us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.159010 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.159315 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.159415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 02:50:05.202432 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.192879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.7.246.124:46559)
0619 02:50:05.193088 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.193095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.200103 (+ 7008us) server_negotiation.cc:407] Connection header received
0619 02:50:05.200647 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.200654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.200844 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.200938 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.202159 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48542: BlockingRecv error: recv got EOF from 127.0.0.1:48542 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:50:05.255359 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.236665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.7.246.125:33259)
0619 02:50:05.236851 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.236857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.247968 (+ 11111us) server_negotiation.cc:407] Connection header received
0619 02:50:05.248499 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.248505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.248600 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.248681 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.249470 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.250664 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.252851 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.253616 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.253749 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.254409 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.254565 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.254645 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.254937 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.255025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 02:50:05.299232 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.289851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.7.246.124:46559)
0619 02:50:05.290050 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.290057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.296987 (+ 6930us) server_negotiation.cc:407] Connection header received
0619 02:50:05.297554 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.297560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.297659 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.297745 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.298991 (+ 1246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48552: BlockingRecv error: recv got EOF from 127.0.0.1:48552 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:50:05.355108 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.338470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.7.246.125:33259)
0619 02:50:05.338668 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.338678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.349414 (+ 10736us) server_negotiation.cc:407] Connection header received
0619 02:50:05.349971 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.349977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.350063 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.350177 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.350622 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.351523 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.353221 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.353757 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.353896 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.354332 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.354457 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.354563 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.354683 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.354762 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:50:05.368647 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.366444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.7.246.124:46559)
0619 02:50:05.366711 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.366720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.367218 (+ 498us) server_negotiation.cc:407] Connection header received
0619 02:50:05.367431 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.367437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.367527 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.367652 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.368411 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48558: BlockingRecv error: recv got EOF from 127.0.0.1:48558 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:50:05.382293 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.375708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.7.246.125:33259)
0619 02:50:05.376037 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.376047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.376633 (+ 586us) server_negotiation.cc:407] Connection header received
0619 02:50:05.376887 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.376898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.377001 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.377143 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.377630 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.378522 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.380057 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.380613 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.380813 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.381248 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.381390 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.381532 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.381823 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.381935 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
W20260619 02:50:05.393941 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.391657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.7.246.124:46559)
0619 02:50:05.391917 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.391923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.392552 (+ 629us) server_negotiation.cc:407] Connection header received
0619 02:50:05.392764 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.392772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.392868 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.392967 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.393724 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48562: BlockingRecv error: recv got EOF from 127.0.0.1:48562 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 02:50:05.411209 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.405179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39508 (local address 127.7.246.125:33259)
0619 02:50:05.405419 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.405441 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.405991 (+ 550us) server_negotiation.cc:407] Connection header received
0619 02:50:05.406234 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.406241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.406334 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.406449 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.406881 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.407718 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.409119 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.409731 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.409892 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.410308 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.410424 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.410550 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.410710 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.410815 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 02:50:05.420428 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.417696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.7.246.124:46559)
0619 02:50:05.417936 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.417942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.418888 (+ 946us) server_negotiation.cc:407] Connection header received
0619 02:50:05.419100 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.419104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.419212 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.419348 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.420165 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48574: BlockingRecv error: recv got EOF from 127.0.0.1:48574 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 02:50:05.445492 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.439394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.7.246.125:33259)
0619 02:50:05.439620 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.439630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.440031 (+ 401us) server_negotiation.cc:407] Connection header received
0619 02:50:05.440233 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.440237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.440315 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.440408 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.440920 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.441796 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.443790 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.444178 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.444275 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.444758 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.444853 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.444915 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.445119 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.445183 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260619 02:50:05.453387 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.451319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.7.246.124:46559)
0619 02:50:05.451535 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.451557 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.451881 (+ 324us) server_negotiation.cc:407] Connection header received
0619 02:50:05.452046 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.452050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.452159 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.452241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.453118 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48582: BlockingRecv error: recv got EOF from 127.0.0.1:48582 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 02:50:05.500522 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.493460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.7.246.125:33259)
0619 02:50:05.493713 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.493720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.494098 (+ 378us) server_negotiation.cc:407] Connection header received
0619 02:50:05.494286 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.494292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.494380 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.494492 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.494913 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.496201 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.498491 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.499185 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.499309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.499670 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.499806 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.499889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.500094 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.500182 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 02:50:05.509806 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.507399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.7.246.124:46559)
0619 02:50:05.507607 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.507613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.508113 (+ 500us) server_negotiation.cc:407] Connection header received
0619 02:50:05.508293 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.508297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.508375 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.508466 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.509427 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48588: BlockingRecv error: recv got EOF from 127.0.0.1:48588 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:50:05.582413 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.577376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.7.246.125:33259)
0619 02:50:05.577674 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.577679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.577911 (+ 232us) server_negotiation.cc:407] Connection header received
0619 02:50:05.578069 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.578073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.578147 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.578243 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.578611 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.579429 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.580649 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.581128 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.581314 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.581785 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.581904 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.581975 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.582113 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.582177 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 02:50:05.591018 31166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:05.589038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.7.246.124:46559)
0619 02:50:05.589235 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.589239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.589768 (+ 529us) server_negotiation.cc:407] Connection header received
0619 02:50:05.589926 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.589930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.590003 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.590088 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:05.590774 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48598: BlockingRecv error: recv got EOF from 127.0.0.1:48598 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:50:05.666558 31183 catalog_manager.cc:1259] Loaded cluster ID: 133417175d274a6ab2ea313369505147
I20260619 02:50:05.666698 31183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: loading cluster ID for follower catalog manager: success
I20260619 02:50:05.669267 31183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: acquiring CA information for follower catalog manager: success
I20260619 02:50:05.671109 31183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c669725a9474d208f4f5b195a9b9d6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:50:05.727977 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.722720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.7.246.125:33259)
0619 02:50:05.722946 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.722954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.723301 (+ 347us) server_negotiation.cc:407] Connection header received
0619 02:50:05.723532 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.723536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.723604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.723706 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.724130 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.724870 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.726330 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.726766 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.726917 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.727309 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.727407 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.727506 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.727645 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.727723 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:50:05.738816 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.733486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.7.246.124:46559)
0619 02:50:05.733719 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.733723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.734047 (+ 324us) server_negotiation.cc:407] Connection header received
0619 02:50:05.734189 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.734193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.734339 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.734439 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.734759 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.735451 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.736735 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.737123 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.737213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.737716 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.738283 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.738418 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.738497 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.738559 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:50:05.752781 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.744472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.7.246.125:33259)
0619 02:50:05.744703 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.744712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.746252 (+ 1540us) server_negotiation.cc:407] Connection header received
0619 02:50:05.746535 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.746541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.746647 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.746717 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.747631 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.748415 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.750803 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.751185 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.751261 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.751988 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.752117 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.752177 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.752398 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.752479 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:50:05.752755 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.745414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.7.246.124:46559)
0619 02:50:05.745642 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.745650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.746800 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 02:50:05.747046 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.747051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.747128 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.747221 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.747867 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.748807 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.750565 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.751201 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.751335 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.751967 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.752092 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.752196 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.752315 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.752439 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:50:05.753232 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.744337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.7.246.126:39341)
0619 02:50:05.745137 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:05.745146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:05.745860 (+ 714us) server_negotiation.cc:407] Connection header received
0619 02:50:05.746220 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:05.746229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:05.746486 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:05.746673 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:05.747329 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.748437 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.750770 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:05.751434 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:05.751578 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:05.752128 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:05.752626 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:05.752736 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:05.752831 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:50:05.752921 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":503,"threads_started":1}
I20260619 02:50:05.757573 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 30948
I20260619 02:50:05.758674 31010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:05.867157 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 30948
I20260619 02:50:05.888165 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 31022
I20260619 02:50:05.889302 31084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:05.996716 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 31022
I20260619 02:50:06.017503 8153 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu pid 31096
I20260619 02:50:06.018697 31158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:50:06.124886 8153 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfqHoa6/build/asan/bin/kudu with pid 31096
2026-06-19T02:50:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4409 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8779 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358903 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST